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Even  if  your  agency  does  not  require  you  to  publish,  consider  tnese  career-enhancing  possibilities: 

•  Share  your  acquisition  research  results  with  the  Acquisit.on,  Technology,  and  Logistics 
(ATScL)  community 

•  Change  the  way  Department  of  Defense  (DoD)  does  business. 

•  Help  others  avoid  pitfalls  with  lessons  learned  or  best  practices  from  your  prc-ject  o:  program. 

•  Teach  others  with  a  step-by-step  tutorial  on  a  process  cr  approach. 

•  Share  new  information  that  your  program  has  uncovered  or  discovered  through  the 
implementation  of  new  initiatives. 

•  Condense  your  graduate  project  into  something  beneficial  to  acquisition  professionals. 
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FROM  THE 
CHAIRMAN  AND 
EXECUTIVE 
EDITOR 

Dr.  Larrie  D.  Ferreiro 


The  theme  for  this  edition  of  Defense 
Acquisition  Research  Journal  is 
“Achieving  Dominant  Capabilities 
through  Technical  Excellence  and 
Innovation,"  which  is  the  theme  for 
the  2015  DAU  Training  Symposium 
present  ed  by  the  Defense  Acquisition 
University  Alumni  Association 
(DAUAA).  The  DA'JA  A  sponsors 
the  annual  Hirsch  Research  Paper 
competition,  and  the  winners  of  the 
award  for  2015  are:  First  Place  “The  Value  of  Training:  Analysis 
of  DAU's  Requirements  Management  Training  Results,”  by 
Charles  M.  Court,  Gregory  B  Prothero,  and  Roy  L.  Wood;  and 
Second  Place  “Increase  Return  on  Investment  of  Software 
Development  Life  Cycle  by  Managing  the  Risk— A  Case  Study,” 
by  William  F.  Kramer,  Mehmet  Sahinoglu,  and  David  Ang.  We 
congratulate  both  teams  of  winners,  wrho  were  selected  from  a 
competitive  field  of  entrants. 

The  “Value  of  Training”  article,  as  the  title  indicates,  posits 
that  classroom  training  of  the  type  conducted  at  the  Defense 
Acquisition  University  noticeably  incieases  a  student's 
learning,  and  at  the  same  time  lays  to  rest  several  long-held 
assumptions  about  differences  in  the  learning  capability  of  dif¬ 
ferent  demographic  groups— inside  versus  outside  the  Beltway, 
time  in  billet,  etc.  The  “Increase  Return  on  [nvestment”  article 
examines  the  use  of  statistical  methods  to  examine  software 
error  rates,  allowing  abetter  estimation  of  the  return  on  invest¬ 
ment  during  the  software  development  life  cycle. 
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Two  other  articles  are  included  in  the  print  and  online  editions 
of  this  issue:  “Manage  Toward  Success— Utilization  of  Analytics 
in  Acquisition  Decision  Making”  by  Sean  Tzeng  and  K.  C.  Chang; 
and  “Does  Your  Culture  Encourage  Innovation?”  by  CDR  Craig 
Whittinghill,  USN,  David  Berkowitz,  and  Phillip  A.  Farrington. 
The  article  “Manage  Toward  Success”  proposes  a  statistical 
methodology  called  Bayesian  analysis  to  orient  the  enormous 
amount  of  acquisition  data  and  evidence  to  support  decision 
making.  “Does  Your  Culture  Encourage  Innovation?”  reports 
the  results  of  a  University  of  Alabama  study  of  the  Department 
of  Defense  culture  at  the  organizational  level,  and  proposes 
changes  to  enable  it  to  communicate  and  act  rapidly,  and 
to  innovate. 

The  paper  “DoD  Comprehensive  Military  Unmanned  Aerial 
Vehicle  Smart  Device  Ground  Control  Station  Threat  Model”  by 
Katrina  M.  Mansfield,  Timothy  J.  Eveleigh,  Thomas  H.  Holzer, 
and  Shahryar  Sarkani  analyzes  the  cybersecurity  vulnera¬ 
bilities  of  handheld  UAV  ground  control  stations  in  order  to 
enhance  their  security  and  operational  environment.  The  full 
version  appears  in  the  online  edition  of  this  Journal  (Issue  73). 

The  featured  book  in  this  issue's  Defense  Acquisition 
Professional  Reading  List  is  Glenn  E.  Bugos's  Engineering  the 
F-4  Phantom  II:  Parts  into  Systems ,  reviewed  by  Lee  Vinsel. 

Finally,  the  Defense  Acquisition  Research  Journal  masthead 
continues  to  evolve.  For  our  Editorial  Board,  we  note  that 
Aude-Emmanuelle  Fleurant  has  departed  her  position,  and 
we  acknowledge  her  contributions  to  the  Defense  ARJ.  At  the 
same  time,  we  welcome  to  the  Board  Dr.  William  T.  Eliason 
from  the  Dwight  D.  Eisenhower  School  for  National  Security 
and  Resource  Strategy  On  our  Research  Advisory  Board,  we 
note  that  Dr.  Nayantara  Hensel  and  Mr.  Brett  B.  Lambert  have 
left  their  positions.  We  wish  them  well  and  thank  them  for 
their  help. 

On  a  personal  note,  I  am  pleased  to  welcome  Dr.  Mary  Redshaw 
to  the  Research  Advisory  Board.  Having  left  her  position  at 
the  Defense  Acquisition  University,  where  among  many  other 
things  she  served  as  the  Deputy  Executive  Editor  of  the  Defense 
ARJ  and  my  right  hand,  she  has  joined  the  faculty  at  the  Dwight 
D.  Eisenhower  School.  I  am  very  glad  to  still  be  able  to  call  on 
her  wisdom  and  experience  when  needed. 
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RESEARCH  AGENDA  2015 


The  Defense  Acquisition  Research  Agenda  is  intended 
to  make  researchers  aware  of  the  topics  that  are,  or  should 
be,  of  particular  concern  to  the  broader  defense  acquisition 
community  throughout  the  government,  academic,  and 
industrial  sectors.  The  purpose  of  conducting  research  in 
these  areas  is  to  provide  solid,  empirically  based  findings  to 
create  a  broad  body  of  knowledge  that  can  inform  the  devel¬ 
opment  of  policies,  procedures,  and  processes  in  defense 
acquisition,  and  to  help  shape  the  thought  leadership  for  the 
acquisition  community. 

Eachissue  of  the  Defense  ARJ  will  include  adifferent  selec¬ 
tion  of  research  topics  from  the  overall  agenda,  which  is  at: 
http://www.dau.mil/research/Pages/researchareas.aspx 

Measuring  the  Effects  of  Competition 

•  What  means  are  there  (or  can  be  developed)  to 
measure  the  effect  on  defense  acquisition  costs 
cf  maintaining  an  industrial  base  in  various 
sectors? 

•  What  means  exist  (or  can  be  developed)  of  mea¬ 
suring  the  effect  of  utilizing  defense  industrial 
infrastructure  for  commercial  manufacture 
in  growth  industries?  In  other  words,  can  we 
measure  the  effect  of  using  defense  manufac¬ 
turing  to  expand  the  buyer  base? 
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•  What  means  exist  (or  can  be  developed)  to 
determine  the  degree  of  openness  that  exists 
in  competitive  awards? 

•  What  are  the  different  effects  cf  the  two  best- 
value  source-selection  processes  (tradeoff*  vs. 
lowest  price  technically  acceptable)  on  pro¬ 
gram  cost,  schedule,  and  performance? 

Strategic  Competition 

•  Is  there  evidence  thar  competition  between 
system  portfolios  is  an  effective  means  cf  con¬ 
trolling  price  and  cos:s? 

•  Does  lack  of  competition  automatically  mean 
higher  prices?  Tor  example,  is  there  evidence 
that  sole  source  can  result  in  lower  overall 
administrative  costs  at  both  the  government 
and  industry  levels,  to  the  effect  of  lowering 
total  costs? 

•  What  are  the  long-term  historical  trends  for 
competition  guidance  and  practice  in  defense 
acquisition  policies  and  practices? 

•  Do  what  extent  are  contracts  being  awarded 
noncompetitive  _y  by  congressional  mandate, 
for  policy  interest  reasons?  What  is  the  effect 
cn  contract  price  and  performance? 

•  What  means  are  there  (or  car_  be  developed) 
to  determine  the  degree  to  which  competitive 
program  costs  are  negatively  affected  bylaws 
and  regulations  such  as  the  Berry  Amendment 
and  Buy  American  Act? 
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The  Value  of 

TRAINING: 


Analysis  of  DAU’s  Requirements 
Management  Training 


RESULTS 


Charles  M.  Court,  Gregory  B.  Prothero,  and  Roy  L.  Wood 


In  response  to  Congress,  the  Defense  Acquisition  University  (DAU)  designed 
and  fielded  a  course  of  study  for  Requirements  Management,  including  a 
1-week  advanced  classroom  course.  While  teaching  this  course,  the  DAU 
faculty  routinely  conducts  pre-testing  and  post-testing  to  assist  the  faculty 
and  students  in  assessing  learning  and  retention.  The  faculty  uses  data  from 
these  tests,  along  with  student  demographics,  to  assess  the  value  of  learning 
the  course  provides  and  to  explore  some  initial  assumptions  about  the  readi¬ 
ness  of  the  workforce  to  learn.  Results  show  agreater  than  30  percent  increase 
in  learning  from  pre-  to  post-test  and  debunk  nearly  all  the  preconceived 
notions  the  university  held  about  the  incoming  students. 
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Every  successful  system  acquisition  begins  with  a  well-thought-out 
set  of  operational  capability  requirements.  The  military  services  have 
always  had  some  sort  of  requirements  generation  process  that  told  the 
armories  and  shipyards  what  to  build  for  the  warfighter.  As  acquisition 
became  more  complex,  expensive,  and  risky,  the  Department  of  Defense 
(DoD)  recognized  the  need  for  a  more  formal  system  of  articulating 
requirements  and  the  importance  of  training  both  the  acquisition  and 
the  requirements  workforces. 


The  Joint  Capabilities  Integration 
and  Development  System 

In  2003,  then-Secretary  of  Defense  Donald  Rumsfeld  initiated  a  for¬ 
mal  DoD-level  requirements  generation  process— the  Joint  Capabilities 
Integration  and  Development  System  (JCIDS).  According  to  Chairman 
of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  3170. 01H,  “The  JCIDS 
process  exists  to  support  JROC  [Joint  Requirements  Oversight  Council] 
and  C  JCS  [Chairman  of  the  Joint  Chiefs  of  Staff]  responsibilities  in 
identifying,  assessing,  validating,  and  prioritizing  joint  military  capa¬ 
bility  requirements”  (C  JCS,  2012).  Within  the  context  of  the  National 
Military  Strategy,  JCIDS  provides  a  process  to  identify  and  assess  the 
capabilities  joint  operational  forces  need  to  meet  future  military  chal¬ 
lenges.  A  capabilities-based  assessment  process  identifies  potential  gaps 
in  warfighting  capability  and  drives  changes  to  doctrine,  organization, 
training,  materiel,  leadership  and  education,  personnel,  facilities,  and/or 
policy  (DOTmLPF-P).  Many  requirements  lead  to  nonmateriel  solutions, 
while  other  requirements  call  for  materiel  solutions.  The  JCIDS  process 
generates  the  requirements  and  the  associated  performance  criteria  for 
those  materiel  solutions.  The  Defense  Acquisition  Management  System 
then  fulfills  those  requirements  and  delivers  the  required  capabilities. 

Articulating  a  new  warfighting  capability  requirement  and  defending 
this  need  through  rigorous  discussion  and  analysis  is  a  nontrivial  under¬ 
taking  for  a  requirements  manager.  A  new  military  requirement  can 
initiate  a  decades-long  acquisition  that  requires  the  investment  of  bil¬ 
lions  of  taxpayer  dollars  to  develop,  manufacture,  and  field.  Requirements 
managers  must  be  able  to  correctly  identify,  document,  and  support  the 
compelling  need  for  any  new  system,  then  be  able  to  work  alongside  their 
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acquisition  counterparts  to  field  the  new  capability.  This  is  a  complex 
undertaking.  In  2007,  Congress  formally  directed  the  DoD  to  train  the 
men  and  women  who  develop  new  requirements  under  JCIDS. 


Requirements  Management  Training 

The  National  Defense  Authorization  Act  (NDAA)  of  2007  man¬ 
dated  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  (AT&L),  in  consultation  with 
the  Defense  Acquisition  University 
(DAU),  to  develop  a  training  program 
to  certify  DoD  personnel  with  the 
responsibility  to  generate  capabil¬ 
ity  requirements  for  major  defense 
acquisition  programs  (NDAA,  2006). 

The  congressional  mandate  called  for 
training  both  military  and  DoD  civil¬ 
ian  managers  charged  with  assessing, 
developing,  validating,  and  prioritiz¬ 
ing  requirements  through  the  JCIDS 
process.  This  broad  definition  covered 
relatively  junior  members  of  the  work¬ 
force  up  to  and  including  4-star  generals 
and  admirals  on  the  JROC  who  ulti¬ 
mately  validate  the  requirements.  This 
mandate  created  a  need  for  a  broad  and 
diverse  training  program  at  several  lev¬ 
els  of  sophistication.  Further,  as  Court 
(2010)  pointed  out,  “no  one  person  does 
all  four  tasks  of  assessing,  developing, 
validating,  and  prioritizing”  require¬ 
ments,  so  the  training  program  would 
also  need  to  address  a  wide  variety  of 
tasks  and  competencies. 


DAU  responded  quickly  to  meet  the  congressionally  imposed  deadline 
to  create  and  deploy  a  requirements  management  certification-training 
curriculum  by  September  30,  2008.  Working  with  AT&L  and  the  Joint 
Staff  Directorate  for  Force  Structure,  Resources  and  Assessment  (J8), 
DAU  developed  two  online  courses  for  requirements  managers  and  a 
1-day  classroom  workshop  for  general  and  flag  officers.  These  courses 
were  very  successful,  and  by  the  end  of  fiscal  year  2008,  the  community 
had  logged  more  than  4,200  course  completions.  In  2010,  DAU  added  a 
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TABLE  1.  REQUIREMENTS  MANAGEMENT  TRAINING  CURRICULUM  j 

CLR  101 

Introduction  to  JCIDS 

ROM  no  _  -  ROM  403 

Core  Concepts  .  .  „„j  Requirements 

for  Requirements  .  Executive  Overview 

Management  ana  Workshop 

RQM  413 

Senior  Leader 
Requirements  Course 

4-6  hours 

24-30  hours  5  days  1  day 

Tailored 

A,  B,  C 

B,  C  C  D  (1-3  Star/SES) 

D  (4-Star/Director  of 
Agency) 

Required  Training  Level  Guidelines 

A 

Contribute  to  the  Requirements  generation  and  capability  development  process  in  various  capacities, 
including:  JCIDS  analysis,  subject  matter  or  domain  expertise,  document  staffing  and  coordination 
and/or  administrative  support— Requirements  Originators  and  Support 

B 

Significantly  involved  with  Requirements  generation  and  capability  development  in  specific 
capacities,  i.e  study  leadership,  planning,  writing,  adjudicating  comments,  and  facilitating  inter- 
organizational  development  and  coordination  of  Requirements  documents— Requirements  Writers 
and  Deveiopers 

C 

Designated  by  organizational  leadership  for  advanced  Requirements  instruction;  Primary  duties 
involve  leadership/supervisory  roles  in  requirements  generation  and  capability  development; 
Organizational  representative  in  pertinent  program  management  and  JCIDS  forums  including  FCB 
Working  Group,  FCB,  JCB,  and  JROC  meetings— Requirements  Supervisors ,  Presenters,  and  Trainers 

D 

GO/FO/SES— Validate  and/or  approve  documents;  provide  senior  leadership  and  oversight  of  JCIDS 
analysis  and  staffing:  enforce  Requirements  standards  and  accountability— Requirements  Vaiidators 
and  Prioritizors 

Note.  SES  =  Senior  Executive  Service;  FCB  =  Functional  Capabilities  Board,  JCB  =  Joint  Capabilities  Board;  GO/FO  =  General  Officer/Flag  Officer. 


Source:  Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS) 
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1-week  Advanced  Concepts  and  Skills  for  Requirements  Management 
(RQM  310)  classroom  capstone  course  to  the  curriculum.  Table  1  shows 
the  requirements  management  curriculum  for  designated  individuals 
as  recently  as  2014. 


Requirements  Management  Training  Curriculum 

Developing  new  courses  for  requirements  management  was  an 
entirely  new  area  for  DAU  training  outside  the  customary  acquisition 
disciplines.  The  effort  demanded  an  intense  effort  from  DAU, 
supported  and  sponsored  by  both  the  AT&L  staff  and 
the  Joint  Staff.  DAU  established  integrated  product 
teams  that  included  warfighter  representa-  \  O' 

tives  to  define  the  basic  competencies 
requirements  managers  need  to  oper-  ^ 
ate  successfully  at  different  levels 
of  responsibility.  The  DAU 
faculty  and  outside  subject 
matter  experts  meticu- 
lously  developed  instruction  to 
meet  these  competencies  across 
the  spectrum  of  requirements 
tasks.  The  faculty  adopted  several 
innovative  assessment  tools  to  help 
DAU  answer  the  question  of  whether 
or  not  the  training,  once  deployed,  would  be 
effective. 


Requirements  Certification  Capstone  Course:  New 
Beginnings  and  Opportunities 

Developing  RQM  310,  the  Advanced  Concepts  and  Skills  for 
Requirements  Management  course  demanded  an  intense,  months-long 
effort  by  requirements  and  acquisition  experts  to  ensure  the  course 
conformed  to  the  requirements  management  competency  model  and 
would  challenge  students  to  reach  higher  levels  of  understanding  and 
performance.  DAU  designed  and  piloted  the  new  1-week  course  and 
rolled  it  out  to  students  in  2010. 

Creating  an  entirely  new  classroom  course  allowed  DAU  to  test  and 
apply  many  new  concepts  and  technologies,  RQM  310  includes  faculty 
discussions,  guest  speakers,  computer  simulations,  and  a  challenging 
student  capstone  exercise.  One  of  the  technology  innovations  in  RQM 
310  was  the  routine  use  of  a  classroom-participation  system.  With 
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this  system,  each  student  uses  a  response  device  that  looks  like  a  small 
remote  control  to  respond  to  questions  and  assessments.  During  the  first 
morning  of  the  class,  students  use  their  response  devices  to  take  a  course 
pre-test  and  review  material  from  the  course’s  online  prerequisites. 
Throughout  the  week,  students  continue  to  use  the  response  device  to 
interact  with  faculty  questions  in  the  lessons.  The  RQM  310  students 
also  use  the  response  devices  in  an  in-class  simulation  to  evaluate  and 
discuss  differences  between  programs  depending  on  their  timeline, 
financial  state.  Service  and  Defense  Agency  priorities,  and  issues  such 
as  a  budget  breach  or  a  failed  operational  test. 

RQM  310  student  demographies.  Both  military  and  civilian  require¬ 
ments  managers  attend  RQM  310.  Students  come  from  the  Pentagon 
as  well  as  from  far-flung  Combatant  Commands  and  field  activities. 
Military  members  bring  current  and  relevant  experience  to  the  require¬ 
ments  generation  process.  Typically,  military  requirements  managers 
come  from  operational  and  warfighting  specialties,  and  complete  a 
requirements  management  tour  between  field  assignments.  However, 
there  is  a  relatively  high  turnover  of  military  personnel  through  require¬ 
ments  management  positions,  bringing  in  new  personnel  with  limited  to 
no  JCIDS  or  acquisition  experience,  thus  creating  a  steady  demand  for 
training.  Civilian  requirements  managers  have  greater  tenure  in  their 
positions,  and  provide  continuity  in  requirements  offices  and  a  ''corporate 
memory”  for  their  organizations. 

Assumptions  about  the  workforce.  Given  the  vastly  different  demo¬ 
graphics  of  the  workforce  who  attend  RQM  310,  initial  expectations  were 
that  incoming  knowledge  and  experience  of  the  students  might  also  be 
vastly  different.  For  example,  the  DAU  faculty  assumed  that  civilian 
requirements  managers,  because  of  their  longer  tenure,  would  be  better 
versed  in  JCIDS  and  acquisition  procedures  than  their  military  coun¬ 
terparts.  Another  commonly  held  belief  was  that  students  working  in 
the  nation’s  capital  or  on  a  combatant  commander’s  staff  would  be  more 
knowledgeable  coming  into  the  course  because  of  more  direct  involve¬ 
ment  in  generating  and  vetting  requirements.  In  addition  to  assessing 
the  overall  value  of  training,  this  study  tested  these  major  assumptions 
about  the  workforce,  and  the  results  are  presented  later  in  this  article. 
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Study  Method 

Participants 

This  study  used  the  data  the  DAU  faculty  normally  collects  in  the 
process  of  executing  each  RQM  310  class.  For  purposes  of  this  study,  the 
data  collected  were  from  the  2013  course  offering.  The  faculty  did  not 
originally  anticipate  using  this  course  pre-test  data  in  a  study,  but  rather 
as  a  review  specifically  to  assist  the  students  in  identifying  their  own 
individual  knowledge  gaps,  and  to  alert  the  faculty  to  particular  areas 
of  knowledge  weakness  in  the  class  as  a  whole.  Educational  research  has 
consistently  shown  that  pre-testing  can  help  increase  student  attentive- 
ness  during  the  course  (Sadhasivam,  2013),  and  aid  in  focusing  both 
students  and  faculty  on  improvement  of  particular  knowledge  gaps  (Blin 
&  Wilson,  1994;  Wetstein,  1998). 

While  DAU  developed  the  assessments  and  data  collection  primarily 
to  improve  learning  outcomes,  the  data  have  been  useful  in  providing 
valuable  insights  into  other  aspects  of  the  training.  The  DAU  faculty 
compares  pre-test  data  to  post-test  data  to  determine  overall  student 
improvement  and  to  assess  the  value  of  learning.  Post-test  data  from 
the  end-of-course  assessment  have  similar,  but  not  identical,  questions 
as  those  on  the  pre-test.  The  faculty  also  analyzed  pre-test  data  in  this 
study  against  student  demographics  to  determine  whether  one  group 
might  be  better  prepared  for  the  advanced  concepts  course. 


The  DAU  faculty  compares  pre-test  data  to 
post-test  data  to  determine  overall  student 
improvement  and  to  assess  the  value  of  learning. 
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Research  Design 

As  noted  earlier,  this  research  used  data  collected  from  a  total  of 
263  students  during  the  normal  execution  of  the  RQM  310  course  in 
2013.  The  data  collected  include  pre-test  and  end-of-course  assess¬ 
ment  scores  collected  with  the  student  response  system.  Questions  on 
the  two  tests  are  similar,  but  not  identical,  and  both  instruments  focus 
on  key  learning  and  competencies  needed  by  requirements  managers 
to  be  effective  in  their  jobs.  All  of  the  students  attending  the  RQM  310 
advanced  course  had  previously  completed  the  two  online  prerequi¬ 
site  courses:  Introduction  to  the  Joint  Capabilities  Integration  and 
Development  System  (CLR  101),  and  Core  Concepts  for  Requirements 
Management  (RQM  110).  These  online  courses  are  self-paced,  computer- 
based  training  that  include  their  own  online  assessments  of  student 
progress  and  understanding.  RQM  110  classes  have  assigned  faculty 
who  are  available  to  answer  questions,  mentor  students  who  might  be 
experiencing  difficulty  in  the  course,  and  otherwise  provide  academic 
or  technical  assistance  the  students  might  need. 

DAU  also  collects  student  demographics  in  the  RQM  310  class  to  help  the 
faculty  better  appreciate  the  level  of  experience  and  exposure  to  identi¬ 
fying,  assessing,  and  formulating  capability  requirements.  Based  on  a 
priori  assumptions  mentioned  earlier,  the  faculty  collects  student  data 
on  each  student's  assignment  at  the  time  he  or  she  attended  the  course, 
their  tenure  in  their  current  billet,  aggregate  experience  working  in  the 
requirements  management  field,  and  how  much  of  each  student's  day- 
to-day  work  content  related  to  managing  requirements.  Table  2  shows 
a  breakdown  of  the  demographic  questions  and  the  granularity  of  the 
answers  collected. 

Analysis  of  Pre-Test  and  Post-Test  Scores 

As  a  first  step  in  this  analysis,  tabulating  and  analyzing  pre-test 
scores  produced  a  mean  score  of  51.6  with  standard  deviation  (s.d.)  of 
12.81.  The  tally  of  end-of-course  scores  showed  substantial  improvement 
with  a  mean  of  80.97  and  s.d.  of  10.68.  A  paired-samples  t- test  showed 
the  improvement  in  scores  to  be  statistically  significant,  t(262)  =  37.173, 
p  <  0.0005.  As  noted  earlier,  many  researchers— and  faculty  practitio¬ 
ners— recognize  that  pre-testing  students  can  help  focus  their  attention 
on  desired  outcomes  and  influence  post-test  outcomes.  According  to 
Kim  and  Wilson  (2010),  “there  can  be  substantial  effects  of  pretest  on 
posttest,  especially  when  the  duration  between  them  is  short,  that  is,  less 
than  a  month”  (p.  755).  Researchers  must  consider  and  compensate  for 
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this  fact  in  a  strict  research  context.  However,  since  the  underlying  pur¬ 
pose  of  the  classes  was  to  improve  student  knowledge  and  retention,  the 
substantial  improvement  in  scores  was  desirable  regardless  of  the  cause. 


TABLE  2.  STUDENT  DEMOGRAPHIC  CATEGORIES  j 

Beltway? 

Time 
in  Billet 

Experience 

Percent 

Requirements 

Work 

Career  Field 

Organization 

Inside 

0-6 

Months 

0-6 

Months 

0-25% 

Require¬ 

ments 

Joint  Staff 

Outside 

6-12 

Months 

6-12 

Months 

25-50% 

Operations 

Service  HQ 
Staff 

12-24 

Months 

1-3  Years 

50-75% 

Acquisition 

Major 

Command 

>24 

Months 

3-5  Years 

75-100% 

Other 

Defense 

Agency 

>  5  Years 

100% 

OSD  Staff 

Other 

Analysis  of  the  Student  Demographics 

As  noted  earlier,  a  number  of  assumptions  about  the  student  demo¬ 
graphics  produced  expectations  among  faculty  for  those  who  might 
perform  better  in  the  class,  and  those  who  might  require  more  assistance 
or  remediation.  During  this  research  process,  the  DAU  faculty  wanted 
to  test  these  assumptions  statistically  to  determine  their  accuracy.  To 
do  so,  the  faculty  tested  each  of  the  assumptions  using  SPSS  t-tests  or 
analysis  of  variation  (ANOVA)  to  examine  the  mean  scores  of  each 
subgroup  on  the  pre-test  data.  The  discussion  below  outlines  the  assump¬ 
tions  and  test  results.  In  short,  almost  none  of  the  entering  assumptions 
proved  to  be  true,  and  the  classes  were  far  more  homogeneous  in  terms  of 
pre-test  performance  without  regard  to  prior  experience  or  assignment. 

Assumption  1.  Students  from  inside  the  (Washington,  DC)  Beltway 
would  be  better  prepared  than  those  in  field  activities  outside  the 
Beltway.  An  independent-samples  t-test  assessed  the  means  of  the 
pre-test  scores  between  the  two  groups.  The  inside-the-Beltway  group 
average  pre-test  score  was  52.28  ±  12.5  and  the  outside-the-Beltway 
group  posted  an  average  score  of  59.79  ±  13.2.  The  t-test  analysis  found 
no  statistically  significant  differences  between  student  groups  at  a  95% 
confidence  level,  £(263)  =  0.93,  p  =  0.473. 
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Assumption  2.  Students  with  more  time  in  their  current  billet  will 
be  better  prepared  than  those  with  shorter  tenures.  The  assessment 
divided  the  students  into  those  with  less  than  6  months  in  their  current 
positions,  those  with  6-12  months,  those  with  12-24  months’  tenure, 
and  those  with  greater  than  24  months  in  the  job.  Since  many  military 
requirements  managers  historically  have  shorter  tours  in  requirements 
billets  between  operational  tours,  observers  could  assume  that  longer 
tenures  might  better  prepare  students  for  the  advanced  course.  The 
analysis  did  not  support  this  assumption,  however.  The  means  of  the 
group  scores  on  the  pre-test  varied  only  between  49.5  and  53.7.  An 
ANOVA  test  on  the  groups  revealed  no  statistically  significant  differ¬ 
ences  in  their  respective  performances  on  the  pre-test,  F(3,  258)  =  1.11, 
p  =  0.344. 


Assumption  3.  Students  with  greater  experience  in  requirements  man¬ 
agement  would  be  better  prepared.  To  test  this  assumption,  the  analysis 
subdivided  the  students  into  groups  with  less  than  6  months'  experi¬ 
ence,  those  with  6-12  months'  tenure,  1-3  years,  3-5  years,  and  greater 
than  5  years.  An  ANOVA  test  on  this  data  did  find  a  single  statistically 
significant  difference  between  groups  of  students  as  determined  by  the 
one-way  ANOVA,  F( 4,  258)  =  3.096,  p  =  0.016.  A  Tukey  post-hoc  test  on 
the  data  revealed  that  students  with  3-5  years  of  experience  showed  a 
statistically  significant  average  higher  score  (56.7  versus  48.2)  on  the 
pre-test  than  less  experienced  students  with  6-12  months'  experience. 

Assumption  4.  Students  who  spend  a  greater  amount  of  day-to-day  time 
working  on  requirements  will  show  better  preparation  for  the  class.  For 
this  test,  the  analysis  divided  the  students  into  five  groups:  (1)  students 
who  reported  working  on  requirements-related  tasks  less  than  25%  of 
the  time;  (2)  those  with  requirements  work  between  25%  and  50%;  (3) 
students  with  requirements  work  from  50%  to  75%;  (4)  those  whose 
requirements  content  in  their  workday  were  between  75%  and  100%;  (5) 
students  whose  work  was  100%  exclusively  related  to  requirements.  The 
ANOVA  analysis  for  these  groups  again  pointed  to  no  statistical  differ¬ 
ences  between  the  pre-test  means,  F(5,257)  =  1.48,  p  =  0.195.  The  pre-test 
average  scores  for  these  groups  varied  only  between  50  and  53.6. 

Assumption  5.  Designated  requirements  managers,  and  perhaps  acqui¬ 
sition  professionals,  will  be  better  prepared  for  the  class.  Here,  the 
demographic  questions  asked  the  students  to  self-identify  their  primary 
career  field:  requirements,  acquisition,  operational/warfighter,  or  other. 
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The  ANOVA  analysis  of  the  mean  pre-tests  scores  for  these  groups  found 
no  statistically  significant  differences,  with  mean  scores  between  48.9 
and  53.6,  F(3, 259)  =  0.880, p  =  0.452. 

Assumption  6.  Organizational  assignment  will  have  some  impact  on 
student  readiness.  The  initial  assumption  was  that  there  might  be  some 
relationship  between  the  student’s  assigned  organization  and  his  or  her 
score  on  the  pre-test.  For  example,  the  faculty  might  expect  a  student 
assigned  to  the  Joint  Staff  or  Combatant  Command  to  do  more  work 


This  analysis  debunked  nearly  every  assumption 
about  factors  that  might  affect  student 
preparedness  for  the  advanced  course. 


directly  or  indirectly  in  creating,  assessing,  or  approving  requirements 
than  students  from  other  organizations.  For  this  analysis,  the  study 
broke  the  student  sample  into  those  who  worked  on  the  Joint  Staff, 
Service  Headquarters  Staff,  major  military  command,  Defense  Agency, 
Office  of  the  Secretary  of  Defense  Staff,  a  Combatant  Commander  Staff, 
or  other.  Once  again,  the  ANOVA  showed  no  statistical  differences  in 
mean  pre-test  scores  of  the  students,  regardless  of  their  assignment,  F(6, 
256)  =  0.312,;p  =  0.930. 

Significance  of  the  Analysis 

This  analysis  debunked  nearly  every  assumption  about  factors  that 
might  affect  student  preparedness  for  the  advanced  course.  Each  of  these 
assumptions  made  sense  on  an  intuitive  level,  and  the  results  have  been 
surprising.  DAU  will  need  to  do  more  work  to  determine  exactly  why 
these  assumptions  were  untrue,  but  preliminary  analysis  offers  two 
potential  explanations.  First,  the  knowledge  of  students  coming  into 
the  course  is  much  more  homogeneous  than  originally  believed.  This 
may  be  the  result  of  all  students  being  required  to  take  the  same  online 
preparatory  courses.  Introduction  to  the  Joint  Capabilities  Integration 
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and  Development  System,  CLR 101,  and  Core  Concepts  for  Requirements 
Management,  RQM 110.  Students  who  take  these  courses  may  come  into 
the  advanced  RQM  310  with  a  common  baseline  of  knowledge  learned 
primarily  from  those  classes.  Another  possibility  is  that  individuals  in 
the  requirements  community  typically  work  only  on  single  or  perhaps 
a  handful  of  tasks  related  to  the  broader  process  of  identifying,  assess¬ 
ing,  validating,  and  prioritizing  joint  requirements.  It  is  unlikely  that 
any  individual  student  would  have  a  deep  knowledge,  based  on  experi¬ 
ence,  across  the  entire  process,  regardless  of  tenure  or  organizational 
assignment.  Thus,  expertise  in  any  narrow  area  may  not  contribute  to 
statistically  higher  scores  on  course  material  that  covers  all  areas. 


Summary  and  Conclusions 

DAU  responded  to  the  congressional  mandate  and  met  the  short 
deadline  to  train  and  certify  requirements  managers  through  a  com¬ 
bination  of  online  and  classroom  courses.  The  success  of  the  initial 
DAU  approach  led  to  student  demand  and  leadership  support  to  expand 
the  initial  requirements  curriculum.  The  most  significant  curriculum 
expansion  was  the  development  of  the  Advanced  Concepts  and  Skills  for 
Requirements  Management  course,  RQM  310. 

Developing  a  new  classroom  course  in  a  different,  nontraditional  area 
of  acquisition  allowed  the  DAU  faculty  to  apply  new  technologies. 
Classroom  simulations  enhanced  traditional  teaching  approaches.  The 
simulations  encouraged  the  exchange  of  ideas.  They  helped  requirements 


This  analysis  has  also  been  a  “myth  buster" for 
a  number  of  sincerely  held  assumptions  about 
the  workforce  and  how  demographic  factors 
influence  RQM  310  student  preparation. 
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managers  from  different  Services  and  Defense  Agencies  recognize  their 
common  problems.  Classroom  participation  devices  encouraged  more 
student  involvement. 

The  success  of  using  classroom-participation  devices  led  the  require¬ 
ments  faculty  to  additional  innovation.  Students  take  a  pre-test  on  the 
first  day  of  class,  and  a  final  exam  post-test  at  the  end  of  the  1-week 
course.  Both  exams  use  classroom-participation  “clickers”  with  the 
exam  questions  projected  on  a  classroom  screen.  By  comparing  the 
results  of  the  pre-test  to  the  results  of  the  post-test,  this  analysis 
has  established  that  statistically  significant  improvements  in  scores 
occur,  leading  us  to  conclude  with  confidence  that  student  learning  was 
taking  place. 

This  analysis  has  also  been  a  “myth  buster”  for  a  number  of  sincerely 
held  assumptions  about  the  workforce  and  how  demographic  fac¬ 
tors  influence  RQM  310  student  preparation.  Almost  universally,  the 
assumptions  have  been  wrong,  and  students  coming  into  the  course 
are  much  more  homogeneous  than  the  faculty  anticipated.  Part  of  the 
homogeneity  could  result  from  all  students  taking  the  same  prerequisite 
courses— CLR 101  and  RQM  110— and  coming  into  the  advanced  RQM 
310  with  a  common  baseline  of  knowledge  learned  from  those  classes. 
Another  possibility  is  that  individuals  in  the  community  work  only 
on  single  or  perhaps  a  handful  of  tasks  related  to  identifying,  assess¬ 
ing,  validating,  and  prioritizing  joint  requirements,  thus  no  individual 
student  has  a  deep  knowledge  across  the  entire  process,  regardless  of 
tenure  or  organizational  assignment.  Expertise  in  a  narrow  area  may 
not  contribute  to  statistically  higher  scores  on  course  material  that 
covers  all  areas. 

Nevertheless,  the  success  of  pre-  and  post-testing  in  RQM  310  has 
encouraged  the  faculty  to  expand  this  approach  to  other  requirements 
courses.  Specifically,  the  faculty  is  investigating  how  to  apply  this 
approach  to  the  online  Core  Concepts  for  Requirements  Management 
course,  RQM  110.  Further,  based  on  the  success  of  RQM  310,  additional 
classroom  courses  at  the  Defense  Systems  Management  College  have 
adopted  the  classroom  simulations  and  the  student-participation  sys¬ 
tem,  and  are  collecting  student  demographics  and  learning  data  to  be 
able  to  continuously  improve  course  content  and  learner  performance. 
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Research  Limitations  and  Future 
Research 

As  noted  earlier,  the  data  collected  from  the  RQM  students  were  pri¬ 
marily  for  the  purpose  of  gauging  the  knowledge  of  the  incoming  students 
and  ensuring  that  the  course  delivered  important  content  in  a  way  that 
was  understandable  and  memorable.  This  analysis  did  not  use  random 
samples  or  experimental  methods  that  would  contribute  to  a  rigorous 
scientific  study.  Future  researchers  may  choose  to  close  these  obvious 
gaps  in  a  more  intentional  way.  In  addition,  post-testing  performed  at 
the  end  of  the  class  does  not  guarantee  the  students  will  remember  the 
information  over  the  long  term.  Future  research  may  wish  to  test  stu¬ 
dents  several  weeks  or  months  after  graduation  and  assess  the  results 
of  knowledge  retention  over  time. 
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Appendix 

The  RQM  310  Class  Schedule 

Table  A1  illustrates  when  the  DAU  faculty  administers  the  pre¬ 
course  assessment  and  the  end-of-course  examination.  The  table  also 
lists  the  course  topics  and  uses  a  color  code  to  illustrate  the  different 
class  activities.  Table  A2  explains  the  color  code. 


TABLE  A1.  RQM  310  DAILY  CLASS  SCHEDULE 


Monday  Tuesday  Wednesday  Thursday  Friday 


8:00 

8:30 

9:00 

9:30 

10:00 


10:30 

11:00 

11:30 

12:00 


12:30 

13:00 


13:30 


14:00 

14:30 
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15:00 

JCIDS 

Simulation 

Examination 

Retest 

15:30 

ICD  Review 
Exercise 

Milestone  A  to 
Milestone  B 

16:00 

SIM  Debrief 

16:30 

Test 

Questions 

Examination 

Results 

' 

17:00 

Note.  AoA  =  Analysis  of  Alternatives;  CDD  =  Capability  Development  Document; 
DOTmLPF-P  =  Doctrine,  Organization,  Training,  Materiel,  Leadership  and  Education, 
Personnel,  Facilities,  and  Policy;  FCB  =  Functional  Capabilities  Board;  FOC  = 

Full  Operational  Capability;  ICD  =  Initial  Capabilities  Document;  IT  =  Information 
Technology;  IS  =  Information  System;  JCIDS  =  Joint  Capabilities  Integration  and 
Development  System;  KPP  =  Key  Performance  Parameter;  KSA  =  Key  System 
Attribute;  MDD  =  Materiel  Development  Decision,  PPBE  =  Planning,  Programming, 
Budgeting,  and  Execution;  ROM  =  Requirements;  SIM  =  Simulation. 


TABLE  A2.  COLOR  CODES  RELATING  CLASS  ACTIVITIES  TO 
TOPICS  IN  TABLE  A1 


Administration 


Examination  or  Examination  Debrief 


Lecture/Discussion 


Guest  Speaker 


Exercise 


Computer  Simulation 


Capstone  Exercise  Presentations 
Course  Wrap-up 
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This  research  article  aims  to  identify  and  introduce  cost-saving 
measures  for  increasing  the  return  on  investment  during  the  Software 
Development  Life  Cycle  (SDLC)  through  selected  quantitative  analyses 
employing  both  the  Monte  Carlo  Simulation  and  Discrete  Event  Simu¬ 
lation  approaches.  Through  the  use  of  modeling  and  simulation,  the 
authors  develop  quantitative  analysis  for  discovering  financial  cost 
and  impact  when  meeting  future  demands  of  an  organization’s  SDLC 
management  process  associated  with  error  rates.  Though  this  sounds 
like  an  easy  and  open  practice,  it  is  uncommon  for  most  competitors  to 
provide  empirical  data  outlining  their  error  rates  associated  with  each 
of  the  SDLC  phases  nor  do  they  normally  disclose  the  impact  of  such 
error  rates  on  the  overall  development  effort.  The  approach  presented 
in  this  article  is  more  plausible  and  scientific  than  the  conventional 
wait-and-see,  whatever-fate-may-bring  approach  with  its  accompanying 
unpleasant  surprises,  often  resulting  in  wasted  resources  and  time. 


Keywords:  discrete  event  simulation  (DES),  Monte  Carlo  simulation  (MCS),  error  or 
defect  rate ,  return  on  investment  (ROIX  software  development  life  cycle  (SDLC) 
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The  science  behind  software  development  in  metric  terms  of  return 
on  investment  (ROI)  is  well  known  and  taught  by  many.  Much  work  has 
been  accomplished  in  this  area  albeit  lacking  details  of  execution  on  a 
real-life  problem  (Ferreira,  Collofello,  Shunk,  &  Mackulak,  2009;  Zhang, 
Kitchenham,  &  Pfahl,  2008;  Zhang,  Kitchenham,  &  Pfahl,  2010).  The  art 
of  software  development  is  a  learned  behavior  and  not  one  with  which 
everyone  becomes  comfortable  due  to  its  intricacies  and  learning  cycle. 
The  same  may  be  said  with  respect  to  software  development  life  cycle 
(SDLC)  management  and  distribution  as  depicted  in  Table  1,  where  the 
different  phases  of  an  SDLC  process,  when  applied,  provide  specific 
inputs  and  expected  outputs. 


TABLE  1.  LIFE  CYCLE  PHASE  FLOW 
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Life  Cycle  Phase  (Process)  Flow 


proceed  to  the  next 
phase  or  recycle 
back  through  the 
current  phase  for 
further  refinement. 


As  with  many  processes,  there  is  a  beginning  point  and  a  delivery 
epoch.  SDLC  methodology  is  no  different.  It  enables  standardization  for 
planning  and  organizing,  and  facilitates  cost  estimation.  Though  there 
are  several  different  models  available,  many  are  tweaked  to  best  fit  the 
current  process  or  a  sequence  of  activities  in  a  software  development 
project.  The  life  cycle  used  in  this  article  (Table  1)  has  nine  phases 
beginning  with  the  requirement  review  and  ending  with  the  deploy¬ 
ment  decision.  As  one  begins  with  the  first  phase  (i.e.,  requirement 
review)  and  moves  right,  software  developers  will  observe,  at  a  mini¬ 
mum,  the  activities  that  must  be  performed  in  the  phase  (keep  in  mind 
this  is  a  high-level  depiction).  Moving  right,  there  is  a  decision  to  be 
made  whether  to 


This  decision  is 
only  one  of  many 
for  the  phases; 
however,  it  might 
be  the  most  cru¬ 
cial.  Not  only  will 
schedule  and  cost 
be  impacted,  but 
phase  errors  will 
drive  substantial 
cost  as  well.  An 
organization  needs 
to  understand  the 
impact,  and  that  is  the  intent  of  this  article— namely  to  show  the  phase 
error  impact  to  the  SDLC,  thereby  reducing  overall  project  manage¬ 
ment  cost  by  improving  the  error  rate. 


Each  phase  will  generate  its  own  success  criteria,  allowing  a  develop¬ 
ment  team  to  anticipate  the  degree  of  success  that  can  be  expected 
throughout  the  life  cycle.  Unfortunately,  as  a  development  team  moves 
through  the  SDLC  process,  it  is  common  to  shift  expected  outputs  to 
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the  right  and  ultimately  into  the  next  phase,  if  only  to  remain  on  track 
regarding  the  end  schedule  or  an  expected  financial  burn  rate.  Ultimately, 
reality  will  set  in  and  a  price  to  be  paid  will  become  readily  apparent, 
whether  it  be  in  the  form  of  a  scheduling  or  financial  disruption. 


This  may  be  even  more  prevalent  when  it  comes  to 
*he  acquisition  of  custom  software.  To  be  bet¬ 
ter  prepared  for  the  impact  of  the  shifting 


deliverables  associated  with  the  SDLC  man¬ 


agement  process,  one  must  understand  the 
intricacies  of  the  process  and  especially  the 
mpacts  associated  with  a  product  that 
is  either  late  or  overbudgeted.  Using 
a  discrete  event  simulation  (DES) 
and  Monte  Carlo  simulation  (MCS), 


as  combined,  may  assist  in  quantifying  a 


scenario  impact.  The  primary  raison  d’etre 


# 


of  this  article  is  to  demonstrate  the  poten¬ 
tial  for  modeling  the  SDLC  management 


?\\  process  and  bring  the  cost-saving  factor 
forward  to  improve  the  ROI  by  employ¬ 
ing  statistical  simulation  techniques. 


Therefore,  the  basis  of  this  article  is  to  bring 
attention  to  the  use  of  modeling  and  simulation 
(M&S)  in  developing  a  quantitative  analysis 
for  discovering  potential  scheduling  and  financial 
ROIs  within  the  parameters  of  meeting  future  demands 
of  an  organization’s  SDLC  management  process. 


.  More  specifically,  the  potential  impact  is  asso- 
f  ciated  with  errors  accruing  and  accumulating 
f  throughout  the  process.  That  being  said,  one 


must  be  mindful  that  the  methods  used  to  com¬ 
pile  this  research  article  rely  equally  upon  the  art  of 
simulation  as  well  as  the  ever-enduring  statistical  and 
mathematical  sciences  behind  the  art  of  simulation.  The 


statistical  and  mathematical  computations  used  a  significant  amount  of 
data  gleaned  from  many  years  of  software  development  experience.  It  is, 
however,  through  these  years  of  experience  with  software  development 
projects  that  we  have  come  to  appreciate  an  SDLC  management  process. 
Likewise,  it  is  also  during  this  process  that  we  have  learned  to  exercise 
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a  degree  of  caution  when  evaluating  those  bidding  to  perform  custom 
software  development,  who  typically  bid  on  the  process  with  some  degree 
of  naivete  that  views  every  requirement,  algorithm,  and  interface  as  a 
nonissue  and  the  work  as  always  straightforward.  Most  of  the  time,  this 
is  not  true,  since  hiccups  invariably  surface  along  the  way,  whether  in 
the  form  of  undefined  requirements  or  bad  test  data.  More  often  than 
not,  unforeseen  events  occur,  which  ultimately  impact  both  schedule 
and  cost  to  the  users'  disadvantage. 


Modeling  and  Simulation 
Methodology  for  a  Case  Study 

To  identify  and  incorporate  software  life-cycle  phases  along  with 
function  point  analysis,  software  managers  ought  to  associate  the  error 
rate  per  phase  with  the  time  distribution  per  phase.  Organizations  per¬ 
forming  standard  unit,  integration,  and  functional  testing  will  likely 
only  remove  approximately  70%  of  defects  during  the  life-cycle  phases 
(Jones,  2008).  This  practice  will  allow  other  defects  to  run  through 
the  life  cycle  until  the  bottleneck  becomes  apparent  in  the  final  test¬ 
ing  phase.  The  model  introduction  takes  this  into  account  and  assists 
with  providing  a  rough  order  of  magnitude  (ROM)  to  the  level  of  effort  a 
program  may  encounter.  In  addition,  the  model  also  provides  an  alter¬ 
native  approach  to  facilitate  ROMs  with  the  appropriate  schedule  and 
additional  resources. 

Computer  M&S,  as  programs  or  networks  of  computers  mimicking  the 
execution  of  an  abstract  model  of  many  natural  systems  from  physi¬ 
cal  and  life  sciences  to  social  and  managerial  sciences,  and  primarily 
engineering,  have  become  an  integral  part  of  digital  experimentation. 
M&S  proves  useful  to  estimate  the  performance  of  complex  engineering 
systems  when  too  prohibitive  for  analytical  solutions.  A  simulation  is 
defined  as  the  reproduction  of  an  event  with  the  use  of  scientific  models. 
A  model  is  a  physical,  mathematical,  or  other  logical  representation  of 
a  system,  process,  or  phenomenon.  Time-independent  static  MCS  and, 
conversely,  dynamic  DES  (to  manage  events  in  real  time  for  engineer¬ 
ing  applications)  have  been  extensively  reviewed  (Sahinoglu,  2013). 
Taxonomy-wise,  simulated  computer  models  maybe  stochastic  or  deter¬ 
ministic,  and  dynamic  or  static,  and  discrete  or  continuous.  Computer 
simulation  has  been  widely  used  in  engineering  systems  to  validate  the 
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effectiveness  of  tentative  decisions  regarding  a  new  plan  or  schedule,  or 
its  outcomes,  without  experiencing  the  actual  conditions,  which  could 
cost  more  resources  or  partial  to  full  destruction  such  as  in  the  simula¬ 
tion  of  the  nuclear  bomb  (Sahinoglu,  2007).  In  a  book  titled  Simulation 
Engineering  by  Jim  Ledin  (2001),  the  author  outlines  his  twofold  purpose 
as  follows: 

i)  Simulation  is  an  approach  that  can  significantly  accelerate 
the  product  development  cycle  and  provide  higher  quality  in  the 
final  system. 

ii)  A  simulation  contains  a  set  of  mathematical  models  of  one 
or  more  dynamic  systems  and  the  interactions  between  those 
systems  and  their  environment,  (p.  1) 

Moreover,  the  Institute  for  Electrical  and  Electronics  Engineers' 
Spectrum  (June  2012)  emphasized  that  the  M&S  effect  is  a  creative 
and  time-saving  topic  of  interest  relevant  to  automotive  engineering  of 
hybrid  vehicles,  finding  solutions  to  treating  nuclear  waste,  upgrading  the 
nuts  and  bolts  of  the  electrical  power  (Smart)  grid,  and  supercomputing 
research,  among  other  areas  (Aoyama,  2012). 

Simulation  Approach 

Table  2  depicts  the  conduct  of  an  error  rate  analysis  within  the 
parameters  of  the  SDLC  management  process.  To  better  depict  the  prob¬ 
ability  distribution,  Table  2  associates  the  probability  distributions  with 
each  phase  of  the  life  cycle.  Keeping  in  mind  a  waterfall  model  is  in  play, 
future  research  may  requre  further  phase  delineation  among  the  many 
attributes  of  the  phases.  Note  that: 

•  There  is  a  need  to  simulate  and  model  error  rates  within  the 
SDLC  process.  Schedules  and  costs  are  impacted. 

•  Many  models,  such  as  waterfall,  Agile,  SCRUM,  RAD,  time- 
box,  and  spiral  development  methodologies  exist  today  and 
could  be  used  (Zhang,  et  al.,  2010). 

•  This  simulation  model  (Table  2)  focuses  on  the  error  rates 
associated  with  waterfall  methodologies. 
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•  In  order  to  determine  cost  per  cycle  and  average  cost  per 
phase  when  using  a  development  rate  consisting  of  function 
point  per  staff  month,  calculate  the  error  rates  per  phase 
and  then  aggregate  with  the  suggested  cost  model. 


TABLE  2.  SOFTWARE  DEVELOPMENT  LIFE  CYCLE  DISTRIBUTION 
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Algorithmic  Step-by-Step  Approach  Using  Statistical 
Random  Number  Generation 

Table  3  depicts  iterations  1-1,000  and  provides  the  details/samplings 
used  in  the  simulation  correlating  the  phases  with  probability  distri¬ 
butions,  the  defect  rates,  repair  rates,  lambda,  mu,  standard  deviation 
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TABLE  3.  FACTS 


Per  Function  Point:  1  Utilization  Factor:  0.8 
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Phase  4 

Phase  5 

Phase  6 

Phase  7 

Phase  8 

Phase  9 

Requirement 

Review 

Design 

Code 

Development 

Code 

Certification 

Security 

Certification 

Acceptance 

Testing 

Package 

Acceptance 

Baseline 

Integration 

Deployment 

Decision 

Lognormal 

Lognormal 

Weibull 

Erlang 

Erlang 

Exponential 

Erlang 

Erlang 

Erlang 

2.62,  7.1 

17.13,  73.35 

28.39,  0.81 

1.36,  3 

1.36,  3 

40.9 

1.36,  3 

1.36,  3 

1.36,  3 

Iteration 

1 

0.3149 

0.4046 

0.6248 

0.0024 

0.0792 

1 

0.0027 

0.0792 

0.0697 

2 

0.3417 

0.4051 

0.0091 

0.1525 

0.057 

1 

0.1333 

0.0119 

0.0712 

3 

0.354 

0.4076 

0 

0.1211 

0.1406 

1 

0.0775 

0.0266 

0.0959 

4 

0.3106 

0.3994 

0 

0.151 

0.0567 

1 

0.0382 

0.081 

0.0653 

5 

0.3404 

0.4035 

0 

0.0737 

0.101 

0.09998 

0.0347 

0.039 

0.0178 

998 

0.3452 

0.3985 

0.9986 

0.1153 

0.0296 

1 

0.0163 

0.0309 

0.0522 

999 

0.35 

0.4053 

0.001 

0.1401 

0.0463 

1 

0.073 

0.1473 

0.0663 

1000 

0.2606 

0.4051 

0 

0.1321 

0.0492 

1 

0.073 

0.1473 

0.0663 

Avg  Defects 
Per  Phase 
(Mean) 

0.3086 

0.4024 

0.2259 

0.0681 

0.0697 

0.9809 

0.069 

0.0687 

0.0704 

2.2636 

5td  Dev 

0.0443 

0.0052 

0.395 

00464 

0.046 

0.0946 

0.0456 

0.0464 

0.0451 

0.7686 

Days  Per 
Phase 

25 

20 

80 

10 

15 

10 

5 

10 

5 

180 

Avg  Defects 
Per  Day 

0.012344 

0.020121 

0.002824 

0.006807 

0.004644 

0.098088 

0.013802 

0.006869 

0.014071 

0.179571 

Avg  Repairs 
Per  Day 

0.01543 

0.025151 

0.00353 

0.008509 

0.005805 

0.12261 

0.017253 

0.008586 

0.017589 

0.224463 

Defect  % 

Per  Day 

6.874 

11.2052 

1.5727 

3.7908 

Aggregate  Lambda  0.1796 
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was  used  along  with  a  180 -day  SDLC  to  determine  defect  rates  per  phase. 
T  hese  were  used  in  the  Java  application  to  simulate  and  provide  input  to 
the  findings  in  Table  4.  Note  the  following: 


TABLE  4.  FINDINGS  AND  EXCEL  SPREADSHEET  RESULTS 
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•  Function  point  count  is  maintained  at  one  function  point  for 
the  life-cycle  period  of  180  days.  With  the  distribution  per 
phase  identified  along  with  the  days  per  phase,  the  Average 
Defects  per  Phase  (ADP)  is  introduced  with  the  summation 
of  the  ADP  to  be  the  average  defect  per  one  function  point. 

•  Next,  the  Average  Defects  per  Day  (ADD)  is  calculated 
by  dividing  the  ADP  by  the  Days  per  Phase.  This  output 
becomes  our  lambda  (A)  in  the  phase  calculation  in  deter¬ 
mining  our  Probability  of  Waiting  (PoW). 

•  The  Average  Repairs  per  Day  (ARD)  is  determined  by  mul¬ 
tiplying  the  ADD  by  our  utilization  factor  of  a  constant  0.8 
(80  percent)  from  best  practices  (Malone  &  Mizell,  2009). 

This  output  becomes  our  mu  (|J),  also  used  in  determining 
the  PoW. 

Results 

Factors  used  to  obtain  results  (Table  3)  follow: 

•  Average  Defects  per  Phase  =  (summation  of  each  phase 
distribution)/iterations 

•  Days  per  Phase  =  variable  set  by  experience 

•  Average  Defects  per  Day  =  (Average  Defects  per  Phase)/ 
(Days  per  Phase) 

•  Average  Repairs  per  Day  -  (Average  Defects  per  Day)/utili~ 
zation  factor. 

To  make  use  of  the  facts  in  Table  3,  a  Java  application  (see  Appendix, 
Java  Source  Code  First  Page)  was  developed  to  conduct  several  thousand 
runs  for  the  simulation  and  ultimately  provide  a  statistical  summary  to 
support  Excel  findings.  The  facts  from  the  spreadsheet  shown  in  Table 
4  were  placed  into  this  homebrewed  java  application  where  the  user  can 
identify  the  inputs,  the  number  of  runs,  and  lastly,  can  run  with  either  a 
single-team  or  a  two-team  simulation. 

Table  4  represents  only  one  screen  shot  with  a  single  distribution,  while 
arbitrarily  using  cost  per  hour  of  $55,  team  size  of  10,  and  work  hours  per 
day  to  equal  8.  One  can  vary  the  cost  factors.  Taking  these  factors  into 
account,  the  cost  formula  in  Equation  (1)  is  as  follows: 
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Total  Cost  =  (Days  to  repair  •  work  hours  per  day  •  team 
size  •  hourly  rate).  (1) 

We  can  begin  to  readily  determine  that  the  errors  per  phase  quickly 
outpace  the  efforts  of  a  single  developer  and  throw  the  schedule  and  cost 
model  far  to  the  right.  However,  by  adding  a  second  development  team  to 
assist  with  the  fixing  of  the  errors  per  phase,  the  cost  and  schedule  are 
only  slightly  impacted  (Malone  &  Mizell,  2009). 

One  can  better  appreciate  the  long-term  impacts  when  dealing  with 
contracts  and  why  the  lower  bid  may  initially  seem  the  best  value;  how¬ 
ever,  with  the  software  development  life  cycle,  this  may  not  be  the  case. 
Improper  preliminary  analysis  and  use  of  resources  could  easily  whirl 
the  schedule  and  cost  into  an  embarrassing  tailspin.  The  core  of  this 
research  precludes  this  handicap. 

Other  findings  and  Excel  spreadsheet  results  highlighted  in  Table  4 
follow. 


•  PoW  is  multiplied  with  the  Days  per  Phase  to  obtain  the 
Days  to  Repair  for  each  specific  team. 

•  Multiple  variables  are  added  to  obtain  realistic  cost  of  soft¬ 
ware  development  teams  (such  as  hourly  rate  of  developers, 
team  size,  and  hours  per  workday). 

•  The  formula  used  for  each  team  is:  Total  Cost  =  (repair  days 
•  work  hours  •  team  size  •  hourly  cost). 

Validation 

Does  the  lowest  dollar  contract  actually  deliver  the  best  value?  This 
is  what  the  research  confirms  positively 

Verification 

Validation  of  error  rates  and  function  point  rates  came  from  Jones 
(2008). 

Outcomes 

Development  teams  can  determine  cost  at  granular  phases  within 
the  SDLC  as  it  pertains  to  error  rates  within  software  development. 
Upon  running  the  simulation,  the  aggregated  results  show  significant 
financial  benefits.  Factors  used  to  obtain  results  are  shown  in  Table  3 
(Malone  &  Mizell,  2009). 
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Conclusions 

The  article  responds  to  the  following  question:  When  required  to 
analyze  best-value  contracts  without  using  a  simulation  model,  does 
the  requestor  actually  obtain  true  cost  by  analyzing  a  single  entity 
to  develop  software  versus  aggregated  cost  (Table  4)  delivered  from 
an  additional  pool  of  resources?  Future  work  along  with  inputs  from 


If  the  errors  are  identified  in  the  early  stages  of  a 
software  development  acquisition,  contracting 
officers  may  be  in  a  better  position  to  avoid  the 
lowest  contract  bid  if  they  understand  where 
proper  resources,  when  applied,  may  actually 
decrease  cost  and  schedule,  thus  delivering  a 
successful  acquisition  and  software  functionality. 


software  development  cost  models  will  go  a  long  way  in  producing  a 
better  understanding  of  the  true  cost  of  software  development  and  why 
there  seems  to  be  a  schedule  shift  as  the  SDLC  runs  through  its  phases. 
This  project  scratches  the  surface  by  showing  that  the  assumption  by 
most  software  developers  that  all  contracts  and  estimates  provided  are 
realistic,  does  not  really  portray  the  impact  of  errors  to  the  schedules, 
which  further  increases  cost.  Some  conclusive  findings  of  interest  are 
outlined  below: 

•  Average  cost  per  phase  with  single  team  to  fix  errors  is  an 
estimated  $628,421.20  with  the  original  summation  of  180 
days  per  phase. 

•  Adding  an  additional  team  to  focus  on  errors,  thereby 
increasing  the  cost  for  labor  for  two  teams,  equates  to 
42.14%  savings.  This  is  readily  discerned  in  the  reduced 
number  of  days  to  fix  the  errors.  In  fact,  the  second  team 
will  cost  an  estimated  $363,633.60  in  labor.  The  overall 
estimated  savings  is  $264,787.60  for  the  cost  of  the  repairs. 

•  Future  and  long-term  analysis  should  focus  on  specific 
methodologies  as  well  as  on  the  coding  language. 
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•  Many  organizations  have  invested  in  the  use  of  the  waterfall 
methodology  and  have  been  slow  to  appreciate  the  potential 
cost  and  schedule  impact  from  error  rates  within  the  mul¬ 
tiple  phases  of  the  SDLC. 

•  This  article  aimed  to  present  a  DES  and  MCS  to  determine 
an  outcome  that  can  be  used  to  improve  a  process  and  cut 
costs.  The  error  rate  analysis  project  has  done  just  that. 

•  Through  lengthy  discussions  about  rates  within  the  MCS 
portion  and  the  impact  on  business  development  systems, 
additional  research  and  refinement  may  be  sought  to  fur¬ 
ther  develop  the  phase  rates  from  within  an  organization. 
Additional  research  will  provide  better  understanding  of 
the  impact  for  long-term  software  development  and  error 
rate  impact. 

•  It  is  hoped  that  this  and  later  work  will  enable  future  profes¬ 
sionals  in  software  development  acquisition  to  establish  a 
more  definite  cost  analysis  when  confronting  quantifiable 
data  such  as  function  points  and  development  languages  to 
give  them  a  better  understanding  of  the  impact  of  develop¬ 
ment  errors  within  the  different  phases  of  the  waterfall 
SDLC. 

An  SDLC  is  a  methodological  process  that  from  a  high  level  can  be  used 
to  determine  schedules  and  costs  and  identify  bottlenecks.  However,  it 
seems  only  recently  that  declining  information  technology  budgets  and 
increasing  delivery  costs  require  us  to  slice  the  life  cycle  into  further 
granularity  to  understand  better  the  cost  and  schedule  impacts.  In  an 
attempt  to  correlate  errors  with  phases  and  cost  to  fix,  a  prevailing 
assumption  is  that  the  cost  of  errors  is  flat.  However,  this  may  not  be  so. 
If  the  errors  are  identified  in  the  early  stages  of  a  software  development 
acquisition,  contracting  officers  maybe  in  a  better  position  to  avoid  the 
lowest  contract  bid  if  they  understand  where  proper  resources,  when 
applied,  may  actually  decrease  cost  and  schedule,  thus  delivering  a  suc¬ 
cessful  acquisition  and  software  functionality. 
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Appendix 


JAVA  SOURCE  CODE  FIRST  PAGE 


//package  negexp; 

//  W.  Kramer 

import  java.awt.*; 
import  java.awt.event.*; 
import  javax. swing.*; 
import  java. util. Random; 
import  java.text.*; 

public  class  NegExp  extends  JFrame  { 

//elements  of  user  interface 
private  JLabei  trialsJLabel; 
private  JLabei  meanJLabel; 
private  JLabei  devJLabel; 
private  JLabei  MuJLabel; 
private  JLabei  BetaServiceJLabel; 
private  JLabei  errorRateJLabel, 
private  JLabei  LambdaJLabel; 
private  JLabei  BetaJLabel, 

private  JLabei  servTimeJLabel; //package  negexp; 
//  W.  Kramer 

import  java.awt.*; 
import  java.awt.event.*; 
import  javax. swing.*; 
import  java. util. Random; 
import  java.text.*; 

public  class  NegExp  extends  JFrame  { 
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TOWARD  SUCCESS 


—Utilization  of  Ana  ytics  in  Acquisition  Decision  Making 


Sean  Tzeng  and  K.  C.  Chang 


Large  information  technology  (IT)  projects  such  as  Defense  Business  System 
(DBS)  acqv  isitions  have  been  experiencing  an  alarming  rate  of  large  cost  over¬ 
runs,  long  schedule  delays,  and  under- delivery  of  specified  capabilities.  There 
are  strict  defense  acquisition  laws/regulations/policies/guidance  with  an 
abundance  of  review  and  oversights,  generating  a  plethora  of  data  and  evidence 
for  project  progress.  However,  with  the  size  and  complexity  of  these  large  IT 
projects  and  sheer  amount  of  project  data  they  produce,  there  are  challenges  in 
collectively  discerning  these  data  and  making  successful  decisions  based  on 
them.  This  research  article  develops  an  analytic  model  with  Bayesian  networks 
to  orient  the  vast  number  of  acquisition  data  and  evidence  to  support  decision 
making,  known  as  the  DBS  Acquisition  Probability  of  Success  (DAPS)  model. 


Keywords:  defense  business  system,  acquisition,  analytics,  evidential  reasoning,  Bayesian 
networks 
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Developing  an  information  technology  (IT)  system  to  meet  organiza¬ 
tional  needs  is  not  a  simple  task.  It  is  often  very  extensive,  taking  a  long 
time  to  realize,  and  more  costly  and  difficult  than  originally  imagined. 
This  is  especially  true  for  large  IT  projects  (over  $15  million).  In  a  2012 
study,  University  of  Oxford  researchers  reported  that,  on  average,  large 
IT  projects  run  (based  on  5,400  IT  projects)  45%  over  budget,  7%  over 
time,  and  are  delivered  with  56%  less  value  (Bloch,  Blumberg,  &  Laartz, 
2012).  The  situation  seems  to  be  even  worse  for  Department  of  Defense 
Business  System  (DBS)  acquisition  programs,  where  the  majority  of 
programs  would  meet  the  University  of  Oxford  researchers'  threshold 
for  large  IT  projects.  A  Government  Accountability  Office  (GAO,  2012) 
report  indicates  that  of  10  Enterprise  Resource  Planning  programs  the 
Department  of  Defense  (DoD)  identified  as  critical  to  business  opera¬ 
tions  transformation,  nine  of  the  programs  were  experiencing  schedule 
delays  up  to  6  years,  and  seven  of  the  programs  were  facing  estimated 
cost  increases  up  to  or  even  over  $2  billion.  This  is  occurring  even  though 
acquisition  laws,  regulations,  policies,  guidance,  independent  assess¬ 
ments,  technical  reviews,  and  milestone  reviews  guide  DBS  acquisition. 

Great  amounts  of  data  and  a  large  number  of  artifacts  are  gener¬ 
ated  during  execution  of  DBS  programs.  A  few  examples  include  the 
Integrated  Master  Schedule  (IMS),  Earned  Value  Management  System 

(EVMS)  Metrics,  Business  Case, 
and  Systems  Engineering  Plan 
(SEP),  as  well  as  Risk  Reports  and 
various  independent  assessments. 
These  data/artifacts  are  commonly 
used  by  decision  makers  at  techni¬ 
cal  reviews  and  milestone  reviews 
as  evidence  of  program  progress  to 
support  their  decisions.  However, 
the  development  and  use  of  evi¬ 
dence  to  support  decisions  has  not 
translated  to  desirable  investment 
outcomes.  This  issue  is  analogous  to 
the  experience  of  other  professional 
disciplines  such  as  intelligence, 
criminal  justice,  engineering,  and 
medical  professions.  In  today's 
Information  Age,  acquisition  and 
availability  of  information  and  evi¬ 
dence  no  longer  represent  the  most 
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challenging  issues.  Often  data/evidence  is  abundant,  but  the  availability 
of  analytical  tools  limits  the  ability  to  figure  out  what  all  the  evidence 
means  collectively  and  how  it  supports  the  hypothesis  being  sought. 
Good  decision  making  requires  not  only  information  and  evidence,  but 
the  inference  and  representation  of  the  evidence  to  support  decision 
making.  Currently,  DBS  acquisition  decision  makers  have  limited  means 
to  aid  them  in  holistically  and  logically  processing  what  all  the  available 
evidence  collectively  indicates  about  a  program,  and  for  using  that  evi¬ 
dence  in  a  structured  manner  to  support  decision  making. 

DBS  Acquisition  Probability  of  Success  (DAPS)  is  the  evidence-based 
analytical  tool  developed  to  help  decision  makers  collectively  draw 
inferences  from  the  abundance  of  available  evidence  produced  during 
the  course  of  DBS  acquisition.  Based  on  observations  and  inferences 
of  evidence,  the  DAPS  model  is  able  to  assess  program  performance  in 
specific  subject  matter  knowledge  areas  and  assess  the  overall  likeli¬ 
hood  for  program  success.  DAPS  is  a  way  ahead  to  support  acquisition 
decision  making,  and  an  initial  step  forward  in  improving  human  under¬ 
standing  and  ability  to  innovate  and  engineer  systems  though  evidential 
reasoning. 


Theoretical  Foundations 

A  brief  discussion  on  the  theoretical  foundations  behind  the  DAPS 
research  is  presented  in  this  section.  Topics  include  evidential  reasoning 
and  knowledge-based  management. 

Evidential  Reasoning 

According  to  Schum  (2001),  evidence  is  described  as  “a  ground  for 
belief;  testimony  or  fact  tending  to  prove  or  disprove  any  conclusion” 
(p.  12).  The  evidence  within  the  framework  of  a  DBS  acquisition  pro¬ 
gram  includes  the  artifacts,  technical  plans,  facts,  data,  and  expert 
assessments  that  will  tend  to  support  or  refute  the  hypothesis  of  pro¬ 
gram  success.  However,  evidence  by  nature  is  incomplete,  inconclusive, 
ambiguous,  dissonant,  unreliable,  and  often  conflicting  (Schum,  2001), 
making  the  decision  process  based  on  the  observations  and  inferences 
of  evidence  a  challenging  and  difficult  endeavor.  Evidential  reasoning 
utilizes  inference  networks  to  build  an  argument  from  the  observable 
evidence  items  to  the  hypothesis  being  sought.  In  the  case  of  DBS  acqui¬ 
sition,  the  DAPS  model  argues  for  the  hypothesis  of  program  success  or 
the  alternative  hypothesis  of  program  failure  based  on  the  observations 
of  evidence. 
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A  Bayesian  network  is  a  graphic  modeling  language  used  in  this  research 
to  build  the  inference  network  for  evidential  reasoning.  Its  basis  is  the 
Bayesian  approach  of  probability  and  statistics,  which  views  inference 
as  belief  dynamics  and  uses  probability  to  quantify  rational  degrees  of 
belief.  Bayesian  networks  are  direct  acyclic  graphs  that  contain  nodes 
representing  hypotheses,  arcs  representing  direct  dependency  relation¬ 
ships  among  hypotheses,  and  conditional  probabilities  that  encode  the 
inferential  force  of  the  dependency  relationship  (Neapolitan,  2003). 

A  Bayesian  network  is  a  natural  representation  of  causal-influence 
relationships  (CIRs),  the  type  of  direct  dependency  relationships  built 
in  the  DAPS  model.  CIRs  are  relationships  between  an  event  (the  cause) 
and  a  second  event  (the  effect),  where  the  second  event  is  understood  as 
a  consequence  of  the  first.  CIRs  are  an  important  concept  of  Bayesian 
networks,  and  reflect  stronger  bonds  than  dependency  relationships, 
which  are  not  causal-based  (Pearl,  1988). 

Knowledge-based  Management 

The  DAPS  model  framework  is  based  on  the  concept  of  knowledge- 
based  acquisition  described  by  the  GAO.  In  the  GAO  (2005)  report  for 
National  Aeronautics  and  Space  Administration  (NASA)  acquisition 
programs,  GAO  recommended  to  NASA,  and  NASA  subsequently  con¬ 
curred,  that  transition  to  a  knowledge -based  acquisition  framework  will 
improve  acquisition  program  performance.  The  GAO  has  also  made  the 
same  recommendation  to  the  DoD  in  other  GAO  reports,  including  the 
GAO  (2011)  report. 

GAO  (2005)  describes  the  knowledge-based  acquisition  as  follows: 

A  knowledge-based  approach  to  product  development 
efforts  enables  developers  to  be  reasonably  certain,  at 
critical  junctures  or  “knowledge  points”  in  the  acquisi¬ 
tion  life  cycle,  that  their  products  are  more  likely  to  meet 
established  cost,  schedule,  and  performance  baselines 
and,  therefore  provides  them  with  information  needed 
to  make  sound  investment  decisions,  (p.  9) 

The  more  knowledge  is  achieved,  the  less  risk  or  uncertainty  the  pro¬ 
gram  is  likely  to  encounter  during  the  acquisition  process.  Sufficient 
knowledge  reduces  the  risk  associated  with  the  acquisition  program 
and  provides  decision  makers  and  program  managers  higher  degrees  of 
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certainty  to  make  better  decisions.  The  concept  ofthe  knowledge -based 
acquisition  is  adapted  in  this  research  and  built  into  the  DAPS  model. 
The  Knowledge  Points  mentioned  in  the  Defense  Acquisition  Guidance 
and  the  GAO  reports  are  called  Knowledge  Checkpoints  in  the  DAPS 
model.  DAPS  also  contains  Knowledge  Areas,  which  are  the  subject 
matter  areas  of  DBS  acquisition  in  the  model,  derived  from  Project 
Management  Institute  (PMI)’s  (2008)  Knowledge  Areas. 


DAPS  Bayesian  Network  Model 

DAPS  is  developed  with  a  Bayesian  network  model  in  the  Netica  soft¬ 
ware  tool  (Norsys,  2010).  By  using  a  Bayesian  network,  DAPS  was  able 
to  construct  a  complex  inference  network  to  measure  the  certainties/ 
uncertainties  in  subject  matter  Knowledge  Areas  and  assess  the  level 
of  success  achieved  at  Knowledge  Checkpoints. 

Model  Topology 

The  DAPS  Bayesian  network  model 
contains  a  three-level  structure,  repre¬ 
senting  the  three  types  of  nodes  in  the 
model.  Three  types  of  static  arcs  also 
represent  the  interrelationships  among 
the  three  types  of  nodes  at  a  point  in 
time,  and  one  type  of  dynamic  arc  rep¬ 
resents  the  temporal  relationships  from 
one  point  in  time  to  another.  The  DAPS 
model  at  the  first  Knowledge  Checkpoint, 

Material  Development  Decision  (MDD), 
is  shown  in  Figure  1.  The  topology 
of  the  top  two  levels— Knowledge 
Checkpoint  and  Knowledge  Areas— is 
repeated  at  each  of  the  15  Knowledge 
Checkpoints.  The  bottom  level  contain¬ 
ing  the  Evidence  Nodes— the  observation 
points  of  the  DAPS  model— varies  at  each 
Knowledge  Checkpoint,  depending  on 
various  evidence  requirements. 
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FIGURE  1.  DAPS  MODEL  AT  MATERIAL  DEVELOPMENT  DECISION 
KNOWLEDGE  CHECKPOINT 
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Table  1  outlines  these  DAPS  model  elements. 

The  complete  DAPS  model  contains  15  Knowledge  Checkpoints.  Each 
Knowledge  Checkpoint  has  one  Knowledge  Checkpoint  Node,  seven 
Knowledge  Area  Nodes,  and  a  number  of  Evidence  Nodes.  The  total  is: 

•  15  Knowledge  Checkpoint  Nodes 

•  105  Knowledge  Area  Nodes 

•  258  Evidence  Nodes 

•  258  KA2E  Arcs 

•  195  KA2KA  Arcs 

•  60  KA2KC  Arcs 

•  98  KA2KAi+l  Arcs 
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TABLE  1.  DEFENSE  BUSINESS  SYSTEM  ACQUISITION  PROBABILITY 
OF  SUCCESS  (DAPS)  ELEMENTS 

Nodes 

•  Knowledge  Checkpoint  Nodes  (KC) 

•  Knowledge  Area  Nodes  (KA) 

•  Evidence  Nodes  (E) 

Static  Arcs 

•  Knowledge  Area  Node  to  Knowledge  Checkpoint 
Node  Arcs  (KA2KC) 

•  Knowledge  Area  Node  to  Knowledge  Area  Node 
Static  Arcs  (KA2KA) 

Dynamic  Arc 

•  Prior  Knowledge  Area  Node  at  the  previous 
Knowledge  Checkpoint  to  the  same  Knowledge 
Area  Node  at  the  next  Knowledge  Checkpoint 
Dynamic  Arcs  (KA2KAi+1) 

Knowledge  Checkpoint  Node.  The  Knowledge  Checkpoint  is  the  top- 
level  node,  which  cumulates  all  information  about  the  DBS  acquisition 
program  at  that  decision  point,  assessing  the  likelihood  of  program 
success.  It  provides  a  cumulative  measurement  of  success  achieved  by 
the  program  up  to  the  current  Knowledge  Checkpoint,  and  is  the  metric 
that  can  be  used  to  help  decision  makers  decide  whether  the  program 
has  demonstrated  enough  certainty  and  maturity  to  move  on  to  the 
next  phase. 

Knowledge  Checkpoints  are  modeled  as  leaf  nodes.  They  have  no  chil¬ 
dren  nodes  and  contain  four  Knowledge  Area  Nodes  as  parent  nodes: 
time,  quality,  cost,  and  scope  Knowledge  Area  Nodes,  which  are  the  four 
measurable  (direct)  Knowledge  Areas  in  the  DAPS  model.  These  CIRs 
on  the  Knowledge  Checkpoint  Node  represent  the  four  direct  measures 
of  success.  Success  is  defined  in  DAPS  as  meeting  program  time,  cost, 
and  quality  goals  from  a  clearly  defined  program  scope.  The  Knowledge 
Area  Nodes  are  further  discussed  in  the  next  section.  Table  2  lists  the  15 
technical  reviews  and  milestone  reviews  modeled  in  DAPS  as  Knowledge 
Checkpoints  (Defense  Acquisition  University,  2013). 

Knowledge  Checkpoint  Nodes  contain  two  states  describing  the  state 
of  the  program:  “Success”  and  “Failure.”  The  probability  of  these  states 
reflects  the  knowledge  (certainty)  and  risk  (uncertainty)  assessment  of 
the  program  at  the  Knowledge  Checkpoint. 

Knowledge  Area  Node.  Knowledge  Areas  are  the  second-level 
node,  which  measures  the  certainty  and  maturity  attained  for  that 
particular  subject  matter  area  of  DBS  acquisition  at  the  Knowledge 
Checkpoint.  Knowledge  Areas  in  DAPS  are  derived  from  the  nine  Project 
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Management  Body  of  Knowledge  (PMBOK)  Knowledge  Areas  (Project 
Management  Institute,  2008),  integrated  with  the  systems  engineering 
elements  of  defense  acquisition.  These  Knowledge  Areas  are  further 
divided  into  the  measurable  (direct)  and  enabling  (indirect)  Knowledge 
Areas.  Measurable  Knowledge  Areas  include  scope,  cost,  time,  and  qual¬ 
ity  subject  matter  areas,  which  directly  affect  the  measures  of  program 
success  in  DAPS.  Enabling  Knowledge  Areas  include  general  manage¬ 
ment,  systems  engineering,  and  procurement  subject  areas,  which  do 
not  directly  affect  the  measure  of  program  success,  but  are  important 
enabling  factors  that  drive  success. 


TABLE  2.  CASE  1  DAPS  MODEL  OUTPUT 

KC 

P(Success) 

Success  Factor 

MDD 

67.4 

2.067484663 

ITR 

67.1 

2,039513678 

ASR 

64  5 

1,816901408 

MSA 

55.8 

1.262443439 

SRR 

56.3 

1.288329519 

SFR 

56.9 

1.320185615 

PreED 

56.4 

1.293577982 

MSB 

55.2 

1.232142857 

PDR 

53.9 

1.169197397 

CDR 

52,8 

1.118644068 

TRR 

51.9 

1.079002079 

MSC 

51,2 

1.049180328 

PRR 

50.8 

1.032520325 

IOC 

50.5 

1.02020202 

FOC 

50.3 

1.012072435 

The  Knowledge  Areas  represent  an  important  aspect  of  the  DAPS  model. 
They  model  the  static  and  dynamic  complex  interrelationships  and 
effects  within  DBS  acquisition  and  combine  the  observations  of  various 
evidence  items  in  the  subject  matter  Knowledge  Area.  The  arcs  among 
the  Knowledge  Area  Nodes  at  a  static  point— the  K  A2K  A  arcs— model  the 
CIR  of  how  knowledge  in  one  Knowledge  Area  affects  knowledge  in  the 
second  Knowledge  Area.  The  KA2KA  relationships  in  DAPS  are  shown 
in  Figure  2,  which  is  extracted  from  the  model  structure  presented  in 
Figure  1.  The  arcs  in  the  KA2KA  structure  are  selected  based  on  the 
expert  knowledge  elicitation  conducted  as  part  of  this  research. 
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FIGURE  2.  KNOWLEDGE  AREA  TO  KNOWLEDGE  AREA  (KA2KA) 


GRAPH  STRUCTURE 


The  dynamic  arcs  from  a  Knowledge  Area  Node  at  the  prior  Knowledge 
Checkpoint  to  the  same  Knowledge  Area  Node  at  the  next  Knowledge 
Checkpoint— the  KA2KAi+l  arcs— model  the  CIRs  of  DBS  acquisi¬ 
tion  through  time.  The  KA2KAi+l  arc  represents  the  knowledge  in  a 
Knowledge  Area  at  a  prior  Checkpoint  influencing  the  knowledge  of  the 
same  Knowledge  Area  at  the  next  Checkpoint.  DAPS  uses  Knowledge 
Area  Nodes  to  model  the  dynamic  effects  in  the  progression  of  knowledge 
during  an  acquisition  project.  Thus,  each  Knowledge  Area  Node  gains 
information  from  the  observations  at  the  current  Knowledge  Checkpoint 
as  well  as  the  information  cumulated  from  prior  Knowledge  Checkpoints. 

Figure  3  provides  an  example  graph  of  the  KA2KAi+l  arcs  in  green 
arrows  from  the  MDD  Knowledge  Checkpoint  to  the  next  Initial 
Technical  Review  Knowledge  Checkpoint. 

The  arcs  from  Knowledge  Area  Nodes  to  Evidence  Nodes— the  KA2E 
arcs— model  the  CIR  of  how  knowledge  affects  the  outcome  observed 
with  the  evidence.  Figure  4  provides  an  outline  of  the  seven  Knowledge 
Areas  and  select  samples  of  the  evidence  grouped  under  each 
Knowledge  Area. 
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FIGURE  3.  KNOWLEDGE  AREA  @  KC1  TO  KNOWLEDGE  AREA  @ 
KC2  (KA2KAi+l)  ARC  EXAMPLE 


Knowledge  Area  Nodes  contain  two  states  describing  the  state  of  the 
Knowledge  level  achieved  in  the  subject  matter  Knowledge  Area:  “Good” 
and  “Marginal”  The  probabilities  of  these  states  reflect  the  knowledge 
'certainty)  and  risk  (uncertainty)  in  the  subject  matter  Knowledge  Area. 

Evidence  Node.  The  third-  and  bottom-level  nodes  are  the  Evidence 
Nodes  in  the  DAPS  model.  Observations  of  Evidence  Nodes  are  entered 
at  this  level  to  drive  inference  for  assessing  a  program’s  probability  of 
success.  The  only  CIRs  for  this  level  are  the  arcs  from  Knowledge  Area 
nodes  to  Evidence  Nodes— the  KA2E  arcs  described  previously. 

Evidence  Nodes  contain  three  states  describing  the  state  of  the  evi¬ 
dence:  “Outstanding,”  “Acceptable,”  or  “Unacceptable.”  In  summary, 
fliese  states  reflect  the  risk  assessment  of  the  program  in  the  specific 
Knowledge  Area.  Outstanding  would  require  no  worse  than  a  “Low-Risk” 
assessment.  Acceptable  would  require  no  worse  than  a  “Moderate-Risk” 
assessment.  Unacceptable  would  require  a  “High-Risk”  assessment 
or  worse.  Since  these  are  the  Evidence  Node  observations,  one  of  the 
states  is  chosen  to  describe  the  real-world  observation  of  the  evidence. 
This  provides  information  to  the  parent  Knowledge  Area  Nodes,  which 
updates  the  belief  in  the  Knowledge  Area. 
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FIGURE  4.  SAMPLE  OF  EVIDENCE  TAXONOMY 
BY  KNOWLEDGE  AREA 
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Model  Summary 

To  summarize  the  model.  Figure  1  shows  the  inference  network 
at  one  static  point.  At  this  point,  Evidence  Nodes  are  observed  in 
accordance  with  the  three  node  states  (Outstanding,  Acceptable,  or 
Unacceptable)  to  provide  information  on  the  assessment  of  the  cer¬ 
tainty/maturity  in  the  seven  Knowledge  Area  Nodes  through  the  K  A2E 
arcs.  The  assessments  are  evaluated  according  to  the  two  Knowledge 
Area  Node  states:  Good  and  Marginal.  The  Knowledge  Area  Nodes  then 
propagate  the  information  according  to  the  KA2K  A  arcs  to  combine  the 
belief,  based  on  the  evidence  observed  under  the  Knowledge  Area,  as 
well  as  the  belief  in  other  Knowledge  Areas  where  a  CIR  relationship 
exists.  Finally  the  Direct  Knowledge  Area  Nodes  provide  informa¬ 
tion  to  the  Knowledge  Checkpoint  Node  to  assess  the  belief  in  the 
Knowledge  Checkpoint  Node  states— Success  and  Failure— through 
the  K  A2KC  arcs,  which  completes  the  information  flow  within  a  static 
point  at  a  Knowledge  Checkpoint. 


Measurable  Knowledge  Areas  include  scope ,  cost , 
time,  and  quality  subject  matter  areas,  which 
directly  affect  the  measures  of  program  success  in 
DAPS. 


The  information  at  the  static  point  within  a  Knowledge  Checkpoint 
is  then  passed  on  to  the  next  Knowledge  Checkpoint  using  the  seven 
Knowledge  Area  Nodes  through  the  KA2KAi+l  arcs,  where  Evidence 
Node  assessment  observations  will  again  be  made.  The  informa¬ 
tion  flow  process  is  then  repeated  14  times  until  the  last  Knowledge 
Checkpoint  Node— the  Full  Operating  Capability  (FOC)  Knowledge 
Checkpoint  Node— is  propagated. 
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DAPS  Decision  Process  and  Case 
Analysis 

DAPS  is  an  analytic  model  that  assesses  program  performance  in 
subject  matter  Knowledge  Areas  and  measures  the  overall  likelihood 
for  success.  Its  basis  is  the  observations  of  evidence  already  being  con¬ 
ducted  through  acquisition  reviews  and  oversight.  DAPS  has  significant 
potential  to  aid  decision  makers  in  holistically  and  logically  processing 
the  mountain  of  evidence  to  support  their  acquisition  decision  making  at 
Knowledge  Checkpoints.  This  section  will  first  discuss  how  DAPS  could 
be  used  in  the  acquisition  process  and  then  demonstrate  its  use  through 
a  case  analysis  and  associated  what-if  analysis. 

DAPS  Support  of  Acquisition  Process 

The  highest  level  of  DAPS  model  output  is  the  probability  of  suc¬ 
cess  measurements  at  the  Knowledge  Checkpoint  Nodes,  based  on  the 
program  knowledge  (certainty)  level  attained.  This  highest  level  DAPS 
model  output  is  the  cumulative  metric  to  support  decision  making  at 
Knowledge  Checkpoints,  aided  by  the  measurements  at  the  second-level 
Knowledge  Area  Nodes. 

Three  alternative  views  are  available  to  the  decision  maker  to  observe 
this  top-level  output  of  DAPS. 


0  P(KC  =  Success)  .  . 

Success  Factor  = - -  (1) 

P(KC  =  Failure) 

First  is  simply  the  probability  of  success  at  the  Knowledge  Checkpoint, 
P(KC  =  Success),  as  outputted  from  the  DAPS  model. 

The  second  alternative  view  is  the  translation  of  the  probability  of 
success  at  Knowledge  Checkpoint  Nodes  into  a  “Success  Factor”— the 
likelihood  ratio  of  Success  over  Failure.  This  view  intends  to  help  deci¬ 
sion  makers  better  comprehend  the  chance  for  success  in  terms  of  ratios, 
illustrating  the  odds  the  program  is  more  likely  to  succeed  than  fail, 
shown  in  Equation  (1). 

The  success  factor  is  presented  in  a  format  similar  to  the  safety  factor, 
which  is  commonly  used  in  engineering  applications  as  a  simple  metric 
to  determine  the  adequacy  of  a  system,  as  well  as  the  widely  used  E  VMS 
metrics  of  the  Cost  Performance  Index  and  Schedule  Performance  Index. 


Defense ARJf  April  2015,  Vol  22  No.  2:192-214 


205 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


A  success  factor  above  1  indicates  that  the  program  is  more  likely  to  suc¬ 
ceed  than  fail,  while  a  success  factor  below  1  indicates  that  the  program 
is  less  likely  to  succeed  than  fail. 

The  third  alternative  view  is  by  the  use  of  adjectival  ratings  (DoD,  2011) 
to  describe  the  Knowledge  Checkpoint  assessment  level.  Table  3  provides 
the  range  of  success  factors  used  for  the  case  analysis,  their  respective 
P(KC  =  Success)  ranges,  their  associated  adjectival  ratings  and  risk  lev¬ 
els,  as  well  as  the  prescriptive  recommended  decisions  for  the  respective 
range  and  rating.  The  ranges  and  ratings  recommended  in  Table  3  reflect 
a  risk  attitude  based  on  heuristics  drawn  from  safety  factor  applications. 
Each  organization  or  decision  maker  would  be  able  to  change  the  ranges 
and  associated  ratings  based  on  their  own  risk  attitude. 


TABLE  3.  KNOWLEDGE  CHECKPOINT  ASSESSMENT  AND 

DECISION  GUIDE 

Success  Factor 

P(KC=Success) 

KC  Assessment 
Level 

Recommended 

Decision 

>9 

>90% 

Outstanding 
(Very  Low  Risk) 

Proceed 

3-9 

75%-90% 

Good  (Low  Risk) 

Proceed 

1.5-3 

60%-75% 

Acceptable 
(Moderate  Risk) 

Proceed  With 
Caution 

0.8-1. 5 

44  4%-60% 

Marginal 
(High  Risk) 

Delay  or 

Corrective  Action 

<0.8 

<44.4% 

Unacceptable 
(Very  High  Risk) 

Corrective  Action 
or  Shut  Down 

In  addition,  the  decision  maker  may  observe  the  predicted  probabil¬ 
ity  of  success  measurements  or  success  factors  at  future  Knowledge 
Checkpoints,  especially  the  Full  Operating  Capability  (FOC)  Knowledge 
Checkpoint— the  final  milestone.  A  success  factor  greater  than  1  at 
FOC,  indicating  that  success  is  more  likely  than  failure  as  the  ultimate 
program  outcome,  would  help  to  support  the  decision  to  proceed.  A 
success  factor  less  than  1,  indicating  that  failure  is  more  likely  than 
success  as  the  ultimate  program  outcome,  would  help  support  the  deci¬ 
sion  for  “Delay,”  “Corrective  Action,”  or  “Shutdown.”  Depending  on  the 
observations  of  evidence,  the  predicted  probability  of  success  at  future 
Knowledge  Checkpoints  may  indicate  a  different  trend  for  success  as 
compared  to  the  assessment  at  the  current  Knowledge  Checkpoint.  It 
provides  an  additional  insight  into  the  program. 
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Case  Analysis 

A  total  of  14  case  analyses  were  conducted  as  part  of  the  DAPS 
research.  Two  of  them  were  conducted  with  a  prototype  Bayesian  net¬ 
work  model  based  on  the  Naval  Probability  of  Program  Success  v2 
framework  (Department  of  the  Navy,  2012)  for  direct  analysis  and  com¬ 
parison.  Twelve  more  case  analyses  were  conducted  on  the  final  DAPS 
model.  One  of  them  is  presented  in  the  discussion  that  follows. 

The  intent  of  this  case  analysis  is  to  test  the  sensitivity  of  the  model  to 
extreme  but  realistic  conditions  and  analyze  the  effect  of  conflicting 
evidence  on  program  success.  The  case  presents  a  hypothetical  program 
where  program  management,  budgeting,  and  funding  support  are  strong, 
along  with  an  outstanding  cost  estimate,  while  contracting/procurement 
actions  are  proceeding  with  adequate  performance.  However,  staffing 
is  determined  to  be  inadequate.  The  program  also  has  not  developed  an 
SEP  or  any  architecture.  Quality  risk  is  high  due  to  the  lack  of  technol¬ 
ogy  maturity.  This  case  is  applied  at  Milestone  A,  and  the  DAPS  model 
is  being  used  to  support  the  Milestone  Decision  Authority  (MDA)’s 
milestone  decision.  The  specific  Evidence  Node  observations  in  DAPS 
appear  in  Table  4. 


TABLE  4.  SPECIFIC  EVIDENCE  NODE  OBSERVATIONS  IN  DAPS 

Acceptable  Business  Case 

Pre-Engineering  Development 
(PreED)  Review 

Unacceptable  Risk  Report 
(Scope)  due  to  no  architecture 
development  to  adequately 
define  the  program  scope 

Unacceptable  manning/staffing 

Unacceptable  (missing)  Systems 
Engineering  Plan 

Outstanding  decisions  outcome 
through  the  Investment  Decision 
Memorandum  (IDM) 

Acceptable  procurement 
progress  and  output— Acceptable 
acquisition  strategy 

Unacceptable  Quality  Risk  Report 
due  to  technology  maturity  issues 

Acceptable  Integrated  Master 
Schedule  (IMS)  and  IMS  progress 
and  Acceptable  schedule  risk 

Outstanding  cost  estimates 

Outstanding  program  charter 

Milestone  Acquisition  Decision 
Memorandum  (ADM)  is  unobserved 
since  decision  has  not  been  made 
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The  model’s  Evidence  Node  observation  inputs  as  well  as  the  Knowledge 
Area  Node  and  the  Knowledge  Checkpoint  Node  results  are  shown 
in  Figure  5.  The  probability  of  success  measure  at  this  Knowledge 
Checkpoint,  as  indicated  by  the  Milestone  A  Knowledge  Checkpoint 
Node,  is  at  55.8%.  This  is  the  result  of  the  model  even  with  only  four 
unfavorable  observations  as  compared  to  12  favorable.  The  program’s 
time  knowledge,  cost  knowledge,  procurement  knowledge,  and  general 
management  knowledge  are  likely  to  be  good;  while  scope  knowledge, 
systems  engineering  knowledge,  and  quality  knowledge  are  likely 
to  be  marginal. 


FIGURE  5.  CASE  ANALYSIS  OUTPUT  AT  MILESTONE  A 
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The  probability  of  success  measurement  at  Milestone  A  is  derived 
from  the  scope,  quality,  time,  and  cost  Knowledge  Area  measurements. 
Although  the  evidence  at  this  Knowledge  Checkpoint  strongly  supports 
that  the  program  has  attained  Good  knowledge  in  the  time  Knowledge 
Area  at  79.6%,  and  in  the  cost  Knowledge  Area  at  99.9%,  the  evidence 
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does  not  support  the  same  argument  for  the  quality  Knowledge  Area 
and  scope  Knowledge  Area,  measured  only  at  41.4%  Good  and  37%  Good, 
respectively.  From  the  elicitation  of  the  expert  knowledge  conducted 
in  the  research,  the  DAPS  model  specified  the  weighted  influences 
of  quality  Knowledge  Area  and  scope  Knowledge  Area  to  be  twice  as 
strong  as  the  weighted  inferential  forces  of  time  and  cost  Knowledge 
Area,  producing  the  55.8%  Success  measurement  for  Milestone  A 
Knowledge  Checkpoint. 

Figure  5  outlines  the  probability  of  success  for  the  case  analysis  at  each 
of  the  15  Knowledge  Checkpoints  and  their  respective  success  factors, 
based  on  the  observation  inputs  at  Milestone  A. 

Based  on  the  success  factor  of  1.26  at  Milestone  A,  the  Knowledge  Level 
of  the  acquisition  program  is  rated  as  Marginal  with  a  recommended 
action  of  Delay  or  Corrective  Action.  The  fact  that  the  future  success 
factors  past  Milestone  A  are  all  above  1  bodes  well  for  this  program, 
however,  indicating  that  the  program  contains  a  solid  foundation  for 
possible  future  success.  Within  the  DAPS  model,  this  can  be  attributed 
to  the  high  general  management  Knowledge  Area  and  cost  Knowledge 
Area  results.  The  general  management  Knowledge  Area  acts  as  the  root 
node  in  each  Knowledge  Checkpoint  instance  computation,  and  has  a 
strong  influence  on  the  other  six  Knowledge  Areas.  The  cost  Knowledge 
Area  is  the  only  leaf  node  within  the  Knowledge  Area  network  structure 
and  is  a  strong  indicator  of  the  adequacy  of  the  other  Knowledge  Areas. 

With  the  “Marginal”  rating  and  recommendation  of  “Delay  or  Corrective 
Action,"  sufficient  evidence  is  not  present  to  either  defend  a  favor¬ 
able  decision  to  proceed  or  unfavorable  decision  to  shut  down  the 
program.  However,  the  predicted  future  success  factors  indicate  there 
are  favorable  observations  of  evidence  supporting  the  likelihood  for 
eventual  success. 

With  the  Marginal  rating  and  recommendation  of  Delay  or  Corrective 
Action,  available  evidence  is  not  sufficient  either  to  firmly  defend  a 
favorable  decision  to  proceed  or  unfavorable  decision  to  shut  down 
the  program.  However,  the  predicted  future  success  factors  indicate 
available  observations  of  evidence  support  the  likelihood  for  eventual 
success.  Based  on  this  DAPS  assessment,  the  MDA  would  be  advised  to 
delay  the  Milestone  A  decision  until  the  SEP  and  architecture  artifacts 
are  adequately  developed.  By  that  time,  the  program  could  be  reassessed 
based  on  the  developed  artifacts  and  the  programs  approach  to  address 
the  staffing  shortage  and  technology  maturity  issues. 
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What-if  Analysis 

Prior  to  the  actual  Milestone  A  Review,  the  program  manager  might 
ask  the  question,  "What  if  the  Milestone  A  Review  were  delayed  beyond 
the  threshold  date  for  a  short  period  in  order  to  develop  the  SEP  and 
the  architecture  to  an  adequate  level?  What  would  that  do  to  my  prob¬ 
ability  of  success  measurement  at  Milestone  A  and  beyond?”  Figure  6 
provides  the  Milestone  A  output  from  DAPS  if  the  SEP  and  the  scope 
risk  level  becomes  acceptable,  while  the  Integrated  Master  Schedule 
(IMS)  Progress  becomes  Unacceptable  due  to  the  missed  Milestone. 
This  "what-if”  scenario  assumes  all  other  observations  of  evidence  for 
this  case  remain  the  same. 


FIGURE  6.  "WHAT  IF"  ANALYSIS  AT  MILESTONE  A 
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Note.  ADM  =  Acquisition  Decision  Memorandum;  GM  =  General  Management;  IDM  = 
Investment  Decision  Memorandum,  IGCE  =  Independent  Government  Cost  Estimate, 
IMS  =  Integrated  Master  Schedule;  KC  =  Knowledge  Checkpoint;  MSA  =  Milestone  A; 
RiskRep  =  Risk  Report;  SE  =  Systems  Engineering;  SEP  =  Systems  Engineering  Plan; 
Strat  =  Strategic. 
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As  shown  in  Figure  6,  if  the  program  manager  worked  to  complete 
the  missing  artifacts  and  delayed  the  Milestone  A  Review  beyond  the 
acceptable  range,  the  probability  of  success  at  Milestone  A  would  have 
been  improved  from  55.8%  to  71.6%,  which  updates  the  success  fac¬ 
tor  from  1.26  to  2.52,  thereby  doubling  it.  A  success  factor  would  have 
changed  the  Knowledge  Level  rating  from  Marginal  to  Acceptable  and 
Recommended  Decision  from  Delay  or  Corrective  Action  to  “Proceed 
with  Caution”  The  significant  change  can  be  attributed  to  two  obser¬ 
vations  of  evidence  being  changed  to  favorable,  while  only  one  is  being 
changed  from  unfavorable  to  favorable:  (1)  the  relative  higher  weight  of 
scope  Knowledge  Area  to  Knowledge  Checkpoint  Success  as  compared 
to  time  Knowledge  Area,  and  (2)  the  overarching  effects  of  systems  engi¬ 
neering  Knowledge  Area  to  the  other  Knowledge  Areas. 

Thus,  if  the  program  manager  delayed  the  Milestone  A  Review  until  the 
SEP  and  the  architecture  were  completed,  the  program  manager  would 
have  provided  the  MDA  better  evidence  to  support  a  favorable  decision 
to  proceed,  as  compared  to  the  original  scenario.  Even  though  falling 
behind  schedule  is  undesirable,  the  what-if  scenario  with  the  Acceptable 
rating  provided  the  MDA  just  enough  proof  of  program  maturity  and 
knowledge  certainty  to  be  allowed  to  Proceed  with  Caution. 


Conclusions 

The  DAPS  model  demonstrated  the  potential  of  an  evidence-based, 
Bayesian  network  model  to  support  acquisition  decision  making.  DAPS 
quantitatively  assesses  a  program’s  likelihood  for  success  by  build¬ 
ing  an  inference  network  consisting  of  observable  quality  evidence, 
intermediate  subject  Knowledge  Areas,  defense  acquisition  Knowledge 
Checkpoints,  and  the  respective  CIRs  among  them.  DAPS  embodies 
the  principles  of  knowledge-based  acquisition  in  its  ability  to  analyze 
DBS  programs’  knowledge  and  certainty  levels  through  the  Knowledge 
Checkpoint  and  Knowledge  Area  measurements.  Through  these  quan¬ 
titative  measures,  DAPS  can  be  used  to  aid  the  acquisition  decisions  at 
Knowledge  Checkpoints,  whether  to  allow  the  program  to  proceed,  delay, 
order  corrective  actions,  or  shut  down  the  program. 

The  DAPS  model  represents  an  initial  step  toward  modeling  and  ana¬ 
lyzing  the  complex  decision  process  for  DBS  acquisition  and  system 
development  projects  in  general.  Future  research  can  be  made  to  expand 
the  Bayesian  network  presented  within  the  DAPS  model,  further  build 
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out  the  underlying  complex  interrelationships  as  well  as  environmental 
effects,  and  further  develop  the  prescriptive  capabilities  to  recommend 
decisions  and  actions.  Potentially  significant  capabilities  and  enhance¬ 
ments  could  be  achieved  when  coupled  with  the  ever-advancing  data 
science  and  computing  power.  Through  the  utilization  of  analytics 
to  represent  the  information  and  evidence  available  and  make  better 
inferences  the  decision  makers  will  be  able  to  arrive  at  better  informed 
decisions,  leading  to  more  successful  programs  and  desirable  invest¬ 
ment  outcomes. 
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Does  Your 

CULTURE  Encourage 

INNOVATION? 


For  many  years  military  leaders  have  been  calling  for  the  U.S.  Armed  Forces  to 
be  more  agile,  adaptive,  and  innovative  in  order  to  defeat  future  and  emerging 
threats.  To  assist  the  military  in  this  endeavor,  the  University  of  Alabama  in 
Huntsville  explored  Department  of  Defense  (DoD)  culture  at  the  organizational 
level.  Having  the  proper  organizational  culture  can  improve  performance  by 
empowering  members  to  interact  better  with  their  environment  to  communicate 


Keywords:  organization,  culture,  military,  innovation,  organic 


and  act  rapidly,  and,  perhaps  most  importantly,  to  innovate.  If  organizational 
culture  does  not  encourage  innovation,  however,  organizations  can  improve 
innovativeness  through  culture  manipulation.  By  implementing  identified 
actions  that  influence  cultural  attributes,  culture  can  be  modified,  and  subse¬ 
quently  organizations  can  improve  innovativeness,  enabling  them  to  meet  new 
and  complex  challenges. 


O  Image  designed  by  Diane  Fleischer 
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Calls  from  Senior  Leadership 

Over  the  past  several  years,  senior  military  leaders  and  DoD  civilians 
have  been  calling  for  more  military  innovation  and  adaptability.  Retired 
Chairman  of  the  Joint  Chiefs  of  Staff  Marine  General  Peter  Pace  called 
on  the  military  to  become  more  adaptive  and  agile  by  applying  “our  expe¬ 
rience  and  expertise  in  an  adaptive  and  creative  manner,  encouraging 
initiative,  innovation,  and  efficiency  in  the  execution  of  our  responsibili¬ 
ties”  (Pace,  2006,  p.  2).  Retired  Navy  Admiral  Mike  Mullen,  also  a  former 
Chairman  of  the  Joint  Chiefs  of  Staff,  stated  that  “new  asymmetrical 
threats  call  for  different  kinds  of  warfighters  ...  smarter,  lighter,  more 
agile  ...  only  by  applying  our  own  asymmetric  advantages— our  people, 
intellect,  and  technology— can  we  adequately  defend  the  nation”  (Mullen, 
2008,  p.  4). 

During  the  Defense  Strategic  Guidance  briefing  held  in  the  Pentagon  on 
January  5,  2012,  President  Barack  Obama,  former  Secretary  of  Defense 
Leon  Panetta,  and  Chairman  of  the  Joint  Chiefs  of  Staff  Army  General 
Martin  Dempsey  introduced  a  new  military  strategy  that  shifts  strategic 
focus  to  the  Pacific  and  Asia.  In  his  remarks,  Panetta  commented  that  the 
military’s  “great  strength  will  be  that  it  will  be  more  agile,  more  flexible, 
ready  to  deploy  quickly,  innovative,  and  technologically  advanced.  That 
is  the  force  of  the  future”  (Panetta,  2012). 

Furthering  a  culture  of  innovation  within  the  DoD  will  contribute  to  the 
achievement  of  these  transformational  visions.  Senior  DoD  leaders  have 
endorsed  and  promulgated  a  culture  of  innovation  dating  back  to  at  least 
2001  when  former  President  George  W.  Bush  challenged  officers  during  a 
speech  at  the  U.S.  Naval  Academy  to  “risk  failure,  because  in  failure,  'we 
will  learn  and  acquire  the  knowledge  that  wrill  make  successful  innova¬ 
tion  possible’”  (Williams,  2009,  p.  59).  Since  his  speech,  DoD’s  culture  of 
innovation  has  improved,  as  evidenced  by  former  Secretary  of  Defense 
Donald  Rumsfeld’s  2006  testimony  to  Congress  during  which  he  stated 
that  the  DoD’s  culture  is  “changing  from  one  of  risk  avoidance  to  a  cli¬ 
mate  that  rewards  achievement  and  innovation”  (Fairbanks,  2006,  p.  37). 

How  can  the  DoD  continue  this  trend?  The  recent  research  has  produced 
some  very  interesting  results  outlined  in  this  article,  on  organizational 
culture,  which  may  provide  at  least  part  of  the  answer. 
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Culture  and  Innovativeness 

Webster's  Ninth  New  Collegiate  Dictionary  defines  culture  as  “the 
customary  beliefs,  social  forms,  and  material  traits  of  a  racial,  religious, 
or  social  group”  (Culture,  1990,  p.  314).  The  DoD's  culture  is  influenced 
heavily  by  its  famous  hierarchical,  mechanistic  organizational  structure. 
Organizational  structure  is  described  as  a  continuum.  A  mechanistic 
structure  is  on  one  extreme  of  the  organizational  system  continuum. 
Typically  mechanistic  structures  have  a  process  where  problems  and 
tasks  are  strictly  defined  via  instructions  and  orders  issued  by  superiors 
who  receive  information  as  it  flows  up  to  them.  Information  follows  a 
vertical  path  up  and  down  the  chain  of  command,  enabling  superiors  to 
maintain  their  command  hierarchy  (Burns  &  Stalker,  1966).  Mechanistic 
structures  (and  cultures)  are  characterized  as  controlled,  formalized, 
and  standardized  (Reigle,  2003),  and  mechanistic  organizations  operate 
to  meet  orders  from  management  to  avoid  mistakes  or  disturbances.  A 
widely  accepted  premise  in  the  research  literature  is  that  a  mechanistic 
structure  can  inhibit  innovativeness  (Beyer  &  Trice,  1978;  Damanpour, 
1991;  Tsai,  Chuang,  &  Hsieh,  2009).  Therefore,  one  can  reasonably 
conclude  that  the  DoD?s  mechanistic  structure  and  culture  would 
inhibit  innovativeness. 

On  the  other  extreme 
of  the  organizational 
system  continuum  is  an 
organic  structure  and 
culture  (Burns  &  Stalker, 

1966).  Organic  struc¬ 
tures  are  believed  to  foster 
innovativeness  (Prakash 
&  Gupta,  2008;  Robbins  & 

Judge,  2009;  Walker,  2007).  These  structures  adapt  to  unstable  condi¬ 
tions  and  change.  They  are  characterized  by  individuals  performing 
their  tasks  outside  of  a  clearly  defined  hierarchy,  considering  their  under¬ 
standing  of  the  workload  of  the  organization  while  accomplishing  their 
tasks.  Control  of  information  flow  no  longer  rests  with  superiors  (Burns 
&  Stalker,  1966).  An  organic  organization  can  operate  flexibly  and  adapt 
quickly  to  a  rapidly  changing  environment  (Jones,  2004).  Organic  cul¬ 
tural  values  encourage  creativity  and  innovation  (Jones,  2004;  Lamore, 
2009),  and  innovative  behavior  (Hartmann,  2006). 
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Fortunately,  for  a  mechanistic  organization  such  as  the  DoD,  some 
organic  subordinate  units  are  possible.  In  fact,  a  blend  of  these  opposite 
structures  can  be  advantageous  to  an  organization.  This  concept  is 
particularly  true  of  organic  structures  operating  within  mechanistic 
structures.  For  example,  units  or  departments  may  have  their  own 
organic  structures,  but  the  overall  culture  of  the  organization  outside 
the  unit  or  department  may  be  influenced  by  its  mechanistic,  formal¬ 
ized  chain  of  command.  Organic  structures  and  cultures  that  exist 
within  a  hierarchical  organizational  structure  improve  performance 
and  enable  development  of  innovations  while  taking  advantage  of  quick 
organization-wide  dissemination  and  implementation  of  those  innova¬ 
tions  (Gresov,  1984, 1989). 

Culture  and  structure  interact  with  each  other,  creating  organizations 
that  either  innovate  well,  implement  innovations  well,  or  achieve  both 
depending  on  the  combination  of  culture  and  structure  type  (Gresov, 
1984;  Prakash  &  Gupta,  2008).  This  idea  that  organic  and  mechanistic 
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culture  and  structure  can  exist  simultaneously,  even  symbiotically,  within 
one  organization  is  demonstrated  daily  by  naval  forces  afloat.  This  concept 
has  been  implemented  for  decades  in  the  Command  by  Negation  construct  in 
which  local  commanders  have  the  freedom  to  conduct  warfare  in  their  speci¬ 
fied  area  of  responsibility  until  guidance  from  the  chain  of  command  above 
redirects  their  efforts.  Command  by  Negation  fosters  initiative  and  innovation, 
particularly  at  the  subordinate  organizational  level  (LeGree,  2004). 

Despite  a  decade's  long  use  of  Command  by  Negation,  the  research  literature 
lacks  empirical  evidence  that  describes  the  relationship  between  an  organi¬ 
zation's  structurally  defined  culture  and  its  proclivity  for  innovation.  This 
study  adds  to  the  literature  and  provides  insight  into  how  an  organization 
can  manipulate  its  culture  to  become  more  innovative.  The  rest  of  this  article 
details  our  data  collection,  analysis,  findings,  and  managerial  insights. 


Data  Collection 

This  study  focused  on  surveying  a  representative  sample  large  enough 
to  provide  statistical  rigor.  The  surveyed  sample  comes  from  a  unique  Navy 
community  of  organizations  that  share  a  common  goal.  Even  though  it  was  not 
one  cohesive  unit,  unity  of  purpose  provided  the  members  of  this  community 
a  common  bond.  This  group  of  professionals  consisted  of  roughly  1,100  indi¬ 
viduals  composed  of  scientists,  engineers,  operators,  trainers,  academics,  and 
requirements  officers. 

The  sample  consisted  of  individuals  who  were  active  duty  Navy  personnel, 
government  civilians,  and  contractors.  Demographics  are  displayed  in  Table 
1,  and  as  can  be  seen,  many  similarities  exist  between  the  sample  and  the 
comparison  demographics. 

Upon  inspection,  the  sample  demographics  more  closely  match  Navy  Officer 
Corps  demographics  than  overall  Navy  demographics,  especially  regarding 
gender  and  the  percentage  of  Caucasians.  This  Navy  community  is  also  repre¬ 
sentative  of  a  group  of  professionals,  especially  scientists  and  engineers.  This 
can  be  seen  both  ethnically  and  by  age  in  Table  1.  These  results  are  expected 
since  the  sample  is  made  up  of  professionals  with  significant  experience, 
closely  matching  percentages  and  trends  from  U.S.  college  graduates  and  the 
college-educated  U.S.  science  and  engineering  labor  force. 
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TABLET. 

STUDY  DEMOGRAPHIC  DATA 
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Males 

84.9% 

84.2% 

84.8% 

50.6% 

74% 

Females 

15.1% 

15.8% 

15.2% 

49.4% 

26% 

Ethnicity 

Native 

American 

2.0% 

4.55% 

0.69% 

0.4% 

1.5% 

African 

American 

3.6% 

18.4% 

8.29% 

6.1% 

5% 

Hispanic 

5.6% 

18% 

6.1% 

5.1% 

3.5% 

Subgroup  Total 

n.2% 

41% 

15.1% 

11.6% 

10.0% 

Asian  Indian 

1.2% 

Asian 
(Far  East) 

5.2% 

Asian 

(Middle  East) 

1.6% 

Asian  (Total) 

8.0% 

5.59% 

3.99% 

6.7% 

Pacific  Islander 

2.4% 

1.04% 

0.33% 

0.3% 

Subgroup  Total 

10.4% 

6.63% 

4.32% 

7.0% 

14% 

Caucasian 

78.5% 

62.6% 

81.1% 

81.4% 

84% 

Age 

Age 

(in 

(in 

years) 

years) 

20-30 

15.1% 

<=29 

6.5% 

11% 

31-40 

20.7% 

30-39 

26% 

27.5% 

41-50 

38.2% 

40-49 

27.6% 

27% 

51-60 

16.3% 

50-59 

23.9% 

21.5% 

61+ 

9.6% 

60+ 

16% 

14.5% 
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Although  the  sample  generally  reflects  of  the  active  duty  Navy,  U.S.  col¬ 
lege  graduates,  and  the  college-educated  U.S.  science  and  labor  force,  it 
is  not  reflective  of  gender  percentages  in  all  three  groups,  notably  in  U.S. 
college  graduates  (over  49%  are  women)  (Kannankutty,  2005).  When 
viewed  holistically  in  Table  1,  however,  the  sample  is  reflective  of  the 
active  duty  Navy,  U.S.  college  graduates,  and  the  college-educated  sci¬ 
ence  and  engineering  labor  force.  The  sample  is  most  reflective,  though, 
of  the  Navy  Officer  Corps  and  the  college-educated  U.S.  science  and 
engineering  labor  force  (Kannankutty,  2005;  National  Science  Board, 
2010;  U.S.  Navy,  2010).  Because  of  the  composition  of  this  sample,  it 
can  broadly  be  considered  a  typical  cross-section  of  the  professionals 
who  constitute  the  DoD. 

Measuring  organizational  culture  can  be  accomplished  through  the 
use  of  surveys  and  questionnaires  (Ashkanasy,  Wilderom,  &  Peterson, 
2000;  Kraut  et  al.,  1996).  Using  self-report  surveys,  in  particular,  offers 
respondents  the  opportunity  to  report  their  own  perceptions  of  reality. 
Rentsch  (1990)  stated  that  behavior  and  attitudes  are  determined  by 
perceptions  of  reality  and  not  objective  reality,  so  recording  respon¬ 
dent  perceptions  instead  of  attempting  to  record  reality  is  appropriate 
(Ashkanasy  et  al.,  2000).  Thus,  it  was  determined  that  using  self-report 
surveys  was  the  preferred  means  of  measuring  organizational  culture 
and  innovative  climate  within  the  DoD.  Therefore,  to  collect  data,  a 
7-point  Likert  scale  survey  was  administered  in  March  and  June  2010 
to  evaluate  perceived  organizational  culture  and  innovative  climate. 

A  quick  note  on  culture  and  climate  is  prudent.  Climate  describes  orga¬ 
nizational  expectations  for  behavior  and  outcomes.  People  respond  to 
those  expectations  by  shaping  their  behavior  to  achieve  positive  results 
like  self-satisfaction  and  self-pride  (Scott  &  Bruce,  1994).  Both  culture 
and  climate  are  associated  with  behaviors  (Denison,  1990),  culture 
being  the  shared  values  and  norms  that  shape  behaviors,  and  climate 
representing  organizational  expectations  that  shape  behavior.  Denison 
(1996)  concluded  that  culture  and  climate  are  a  common  phenomenon 
and  that  each  describes  organizational  social  context.  Culture  and 
climate  research  should  be  integrative  and  not  mutually  exclusive 
(Denison,  1996). 

To  conduct  this  research,  a  sample  of  251  individuals  was  obtained  by 
administering  the  Perceived  Organizational  Culture  and  Innovative 
Climate  Assessment  Tool  (POCalCAT),  a  survey  developed  specifically 
for  this  research.  A  thorough  review  of  the  literature  was  conducted  to 
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find  instruments  for  use  that  measure  organizational  culture  (along  the 
organic  and  mechanistic  continuum)  and  innovative  climate.  Twenty- 
four  candidate  survey  instruments  were  identified.  Eleven  of  these 
surveys  measure  organizational  culture  and  13  measure  organizational 
innovative  culture  or  climate  (Whittinghill,  2011).  The  POCalCAT 
Revision  A  was  developed  by  combining  two  valid  and  reliable  Likert 
scale  surveys.  Surveys  combined  were  the  Organizational  Culture 
Assessment  (Reigle,  2003),  which  measures  organizational  culture, 
and  the  Climate  for  Innovation  Measure  (Scott  &  Bruce,  1994),  which 
measures  innovative  climate. 

Reliability 

The  researchers  used  Principal  Component  Factor  Analysis  to 
produce  principal  components,  which  were  used  to  create  a  scale  with 
items  that  reflected  the  construct  being  measured.  The  test  of  reli¬ 
ability  used  was  Cronbach’s  alpha  (Cronbach,  1951).  Cronbach’s  alpha 
is  regarded  as  the  lower  bound  on  reliability  for  a  set  of  congeneric 
measures  (Bollen,  1989).  It  assumes  each  of  the  items  within  the  scale 
contributes  equally  to  the  underlying  trait  (Zeller  &  Carmines,  1980). 
The  alphas  are  reported  in  Table  2. 


TABLE  2.  RELIABILITY  DATA  FOR  POCalCAT  REVISION  A 

Principal  Component 

Cronbach’s  Alpha 

Support  for  Innovation 

0.95 

Workforce  Autonomy 

0.808 

Collaboration 

0.807 

Managerial  Trust/Workforce 
Enthusiasm 

0.774 

Resource  Supply  for  Innovation 

0.555 

As  indicated  by  the  reliabilities,  the  measures  are  relatively  homoge¬ 
neous  for  the  construct  they  purport  to  measure.  Typically,  reliabilities 
greater  than  0.70  are  considered  adequate  for  measurement  analysis 
(Nunnally,  1978).  All  but  one  measure  in  our  analysis  met  this  stan¬ 
dard.  Resource  Supply  for  Innovation  had  a  Cronbach  alpha  score  of 
0.555.  This  score,  however,  is  sufficient.  Cronbach’s  alpha  values  at  or 
above  0.50  have  been  cited  as  acceptable  for  research  (Caplan,  Naidu, 
&  Tripathi,  1984;  Nunnally,  1967;  Pedhazur  &  Schmelkin,  1991).  The 
POCalCAT  Revision  A  also  demonstrated  face,  content,  and  construct 
validity  (Whittinghill,  2011). 
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Sample  Size 

A  sample  size  of  251  was  found  to  be  large  enough  to  provide  sta¬ 
tistical  significance  to  this  study.  The  single-sample  ttest,  Analysis 
of  Variance  (ANOVA),  and  linear  regression  were  used  throughout  the 
research.  First,  for  the  single-sample  ttest,  a  sample  size  of  251  allowed 
a  5%  alpha,  80%  power,  and  0.251  effect  size  level  for  the  statistical 
analysis.  An  effect  size  of  0.251  is  within  the  small  (0.2)  to  medium 
effect  (0.5)  size  range  for  the  t  test  (Cohen,  2009).  For  ANOVA,  seven  of 
11  organizations  surveyed  produced  enough  responses  to  average  34  per 
organization,  resulting  in  statistical  analysis  conducted  at  the  5%  alpha, 
83%  power,  and  medium  effect  (0.25)  size  level  (Cohen,  2009).  Finally,  for 
linear  regression  a  sample  size  of  251  produced  an  alpha  of  5%,  power  of 
80%,  and  effect  size  of  0.175  for  statistical  analysis.  An  effect  size  of  0.175 
is  within  the  small  (0.10)  to  medium  effect  (0.3)  size  range  for  simple 
linear  regression  (Cohen,  2009). 

Before  proceeding,  a  brief  discussion  on  the  concept  of  effect  size  is 
offered.  Cohen  (2009,  p.  9)  indicates  that  an  effect  size  is  “the  degree  to 
which  the  phenomenon  is  present  in  the  population”  or  “the  degree  to 
which  the  null  hypothesis  is  false.”  Therefore,  if  the  null  hypothesis  is 
true,  then  the  effect  size  for  the  treatment  is  zero.  So  if  a  null  hypothesis  is 
false,  it  is  false  to  some  degree,  or  effect  size  (a  nonzero  value).  The  larger 
this  value  is,  the  larger  the  degree  of  manifestation  of  the  phenomenon. 
Larger  sample  sizes  are  needed  to  detect  a  smaller  effect.  According  to 
Cohen  (2009,  p.  25),  a  small  effect  size  is  applicable  for  new  research  areas 
because  in  new  research  areas  where  “the  phenomena  under  study  are 
typically  not  under  good  experimental  or  measurement  control  or  both 
...  the  influence  of  uncontrollable  extraneous  variables  makes  the  size 
of  the  effect  small  relative  to  these.”  A  medium  effect  size  is  defined  as 
“one  large  enough  to  be  visible  to  the  naked  eye.  That  is,  in  the  course  of 
normal  experience,  one  would  become  aware  of  an  average  difference  ... 
between  members  of  professional  and  managerial  occupational  groups 
(Super,  1949,  p.  98)”  (Cohen  2009,  p.  26).  Although  this  research  is  being 
conducted  in  a  relatively  new  research  area,  consistent  dissemination  of, 
and  response  to,  a  reliable  and  valid  Likert-scale  survey  amongst  profes¬ 
sional  and  managerial  groups  led  us  to  determine  an  effect  size  in  the 
small  to  medium  range  was  appropriate.  A  sample  size  of  251,  therefore, 
was  large  enough  to  produce  statistically  significant  results. 
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Analysis 

The  primary  research  question  being  addressed  in  this  study  was  "Is 
there  a  relationship  between  the  perceived  organizational  culture  and 
innovative  climate  of  this  Navy  community?"  To  answer  this  question, 
a  hypothesis  was  formulated:  that  there  is  a  linear  relationship  between 
the  perceived  organizational  culture  and  the  innovative  climate  of  this 
Navy  community.  Linear  regression  was  used  to  test  the  hypothesis. 
Before  proceeding  further,  however,  it  is  appropriate  to  note  that  with 
a  sample  size  of  251,  the  central  limit  theorem  (i.e.,  the  sampling  distri¬ 
bution  approaches  normality  as  sample  size  increases)  applies,  and  a 
normal  population  distribution  was  assumed  (Sheskin,  2004). 

Parametric  statistical  analysis  (i.e.,  single-sample  t  tests  supported  by 
the  nonparametric  Wilcoxon  signed-ranks  tests,  ANOVA,  and  Tukey's 
honestly  significant  difference  [HSD]  tests)  performed  between  orga¬ 
nizations  produced  results  that  indicated  a  correlation  exists  between 
an  organization's  perceived  organizational  culture  and  its  perceived 
innovative  climate. 

To  validate  these  findings,  simple  linear  regression  analysis  of  the  data 
was  conducted.  This  portion  of  the  research  sought  to  determine  whether 
a  relationship  exists  between  organizational  culture  and  innovative  cli¬ 
mate  within  the  surveyed  Navy  community.  For  one  independent  factor 
(degree  of  organic/mechanistic  culture),  an  effect  size  of  0.1  (considered 
small  for  simple  linear  regression),  an  alpha  value  of  5%,  and  a  power 
of  80%  simple  linear  regression  analysis  requires  783  results  for  sta¬ 
tistical  rigor.  However,  this  was  not  achievable  for  the  surveyed  Navy 
community,  so  a  medium  effect  size  (0.3  for  simple  linear  regression) 
was  deemed  sufficient  as  previously  rationalized.  The  medium  effect 
size  (0.3)  was  then  used  to  determine  a  required  sample  size.  According 
to  Cohen,  only  85  results  are  required,  so  the  sample  achieved  provided 
a  range  of  small  to  medium  effect  size  (Cohen,  2009). 

In  this  research,  7-point  Likert-scale  data  were  considered  interval 
data  and  analyzed  with  parametric  statistical  tests  vice  ordinal  data 
analyzed  with  nonparametric  statistical  tests.  This  approach  was 
appropriate  since  the  robustness  of  parametric  tests  and  their  use  with 
ordinal  data  were  supported  in  literature  (Labovitz,  1967;  Norman,  2010). 
Additionally,  it  was  appropriate  to  consider  data  from  the  POCalCAT 
Revision  A  to  be  interval-level  data  since  the  data  are  in  7-point  Likert- 
scale  format  (Boone  &  Boone,  2012);  the  POCalCAT  Revision  A  is 
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both  valid  and  reliable  as  shown  through  Principal  Component  Factor 
Analysis;  and  normality  is  assumed  through  the  central  limit  theorem 
(Allen  &  Seaman,  2007).  Additionally,  nonparametric  tests  were  used 
to  validate  the  parametric  tests  in  this  research,  further  demonstrating 
that  the  results  are  robust. 

Regression  analysis  was  conducted  to  quantify  the  relationship  between 
perceived  organizational  culture  (i.e.,  the  independent  variable)  and 
perceived  innovative  climate  (i.e.,  the  dependent  variable  or  response). 

Results  produced  substantial  evidence  that  a  statistically  significant 
relationship  existed  between: 

1.  The  degree  to  which  an  organization  perceives  itself  to  be 
organic;  and 

2.  The  degree  to  which  it  perceives  itself  to  be  innovative. 

Table  3  shows  that  this  regression  analysis  was  significant  because  the 
regression  analysis  p-value  (<0.5%)  was  less  than  the  accepted  level  of 
significance  (5%),  indicating  the  null  hypothesis— that  the  slope  of  the 
regression  line  is  zero— can  be  rejected,  and  therefore  conclude  that  a  lin¬ 
ear  relationship  exists  between  the  predictor  and  response  (Montgomery, 
Peck,  &  Vining,  2006).  Also,  the  lack  of  fit  p-value  is  greater  than  the 
accepted  significance  level  of  5%,  indicating  that  the  null  hypothesis  (the 
model  is  linear)  cannot  be  rejected  (Montgomery  et  al.,  2006). 


TABLE  3.  REGRESSION  ANALYSIS  RESULTS 


Perceived  Innovative  Climate  Score  =  1.14  +  0.706 
(Perceived  Organizational  Culture  Score) 


Regression 

p-va!ue 

Lack  of  Fit 
p-va!ue 

R2 

R 2  Adjusted 

<0.005 

.413 

48.4% 

48.2% 

Further,  the  coefficient  of  determination  values  R2  and  R 2  Adjusted 
indicate  that  the  model  explains  over  48%  of  the  variance  of  the  data,  so 
over  48%  of  the  variation  of  the  dependent  variable  can  be  explained  by 
the  independent  variable  (Downing  &  Clark,  1997).  This  means  that  over 
48%  of  the  variation  in  perceived  innovative  climate  can  be  explained 
by  perceived  organizational  culture.  Further  interpreting  this  score 
was  rather  subjective,  but  the  closer  the  score  is  to  100%  the  better. 
Explaining  over  48%  of  the  variance  of  the  data,  then,  could  be  improved, 
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but  an  R 2  Adjusted  value  of  48.2%  (from  Table  3)  is  a  sufficient  score  for 
this  study.  Devore  (1995)  stated  that  the  square  root  of  the  coefficient  of 
determination  (or  correlation  coefficient  R)  indicates  strong  correlation 
between  variables  when  this  value  is  greater  than  or  equal  to  0.8  and  less 
than  or  equal  to  1;  medium  correlation  when  this  value  is  greater  than 
0.5  and  less  than  0.8;  and  weak  correlation  when  this  value  is  less  than 
or  equal  to  0.5.  The  square  root  of  the  coefficient  of  determination  (. R 2 
Adjusted)  for  this  regression  model  is  0.694,  indicating  a  medium  level 
of  correlation  (or  degree  of  linear  relationship)  between  variables.  For 
initial  research,  this  is  acceptable.  Further,  the  assumptions  of  normality 
of  the  residual  data,  homogeneity  of  variance,  and  independence  of  the 
data  were  evaluated  and  none  was  violated  (Whittinghill,  2011). 


The  data  suggest  that  organizations  can  improve 
innovativeness  through  culture  modification. 

The  discovered  relationship  revealed  that  the  more  organic  an  organiza¬ 
tion  perceived  itself  to  be,  the  more  it  perceived  itself  to  be  innovative. 
Therefore,  the  data  suggest  that  organizations  can  improve  innovative¬ 
ness  through  culture  modification.  However,  to  accomplish  this,  an 
organization  must  understand  which  attributes  to  develop  in  creating  a 
more  organic  culture  and  subsequently  a  more  innovative  organization. 

The  literature  review  provided  supporting  evidence  that  the  principal 
components  previously  identified  were  the  attributes  that  can  be  modi¬ 
fied  to  create  a  more  organic  culture  and  innovative  climate.  From  the 
literature  review,  27  attributes  were  found  that  contribute  to  innovative¬ 
ness.  This  was  a  large  number  of  attributes  to  study,  and  they  needed  to 
be  reduced  to  a  more  manageable  size.  Initially,  the  27  attributes  were 
evaluated  for  adequacy  and  similarities,  with  19  of  the  attributes  deemed 
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appropriate  for  further  study  (Whittinghill,  2011).  These  19  attributes 
share  some  commonalities,  so  like  attributes  were  grouped  together  and 
placed  in  broader  attribute  categories  (Whittinghill,  2011). 

Whittinghill  identified  five  attributes: 

1.  Support  for  Innovation.  This  is  an  organization’s  encour¬ 
agement  of  creativity  and  willingness  to  change.  It  entails 
communicating  the  importance  of  creative,  innovative 
thinking  and  recognizing  innovators.  Of  all  the  attributes, 
this  one,  according  to  a  review  of  the  research  literature,  is 
most  closely  related  to  an  organization’s  affinity  for  innova¬ 
tiveness  (Ashkanasy  et  al.,  2000;  Scott  &  Bruce,  1994). 

2.  Resource  Supply  for  Innovation.  This  is  defined  as  having 
time,  manpower,  and  funding  available  to  pursue  innovative 
endeavors. 

3.  Collaboration.  This  is  defined  as  a  high  rate  of  interaction 
among  organization  members.  It  is  encouraged  by  valuing 
all  organization  members’  thoughts  and  ideas,  and  by  having 
open  door  policies. 

4.  Workforce  Autonomy.  This  is  defined  as  having  the  flexibil¬ 
ity  to  approach  problems  the  way  an  organizational  member 
sees  fit  based  on  available  information,  free  from  group- 
think,  and  not  overly  impeded  by  regulations. 

5.  Managerial  Trust/Workforce  Enthusiasm.  This  is  best 
described  as  a  workforce  motivated  by  their  work  and 
trusted  to  perform  their  work  without  being  micromanaged. 

Note  that  Principal  Component  Factor  Analysis  revealed 
a  correlated  relationship  between  managerial  trust  and 
workforce  enthusiasm,  so  these  attributes  were  combined 
into  one. 

These  five  attributes  contribute  to  an  innovative  climate  (Ashkanasy 
et  al.,  2000;  Burns  &  Stalker,  1966;  Damanpour,  1991;  Kenny  &  Reedy, 
2006;  LeGree,  2004;  Ruiz-Moreno,  Garcia-Morales,  &  Llorens-Montes, 
2008;  Prakash  &  Gupta,  2008;  Robbins  &  Judge,  2009;  Roxborough,  2000; 
Walker,  2007).  Of  these  five,  support  for  innovation  best  represents  an 
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innovative  climate  because  it  most  directly  influences  organizational 
expectations  for  innovative  behavior.  Expectations  influencing  behavior 
are  fundamental  to  the  definition  of  climate  (Scott  &  Bruce,  1994). 

The  workforce  autonomy,  collaboration,  and  managerial  t rust/ workforce 
enthusiasm  attributes  together  determine  where  on  the  organic/ mecha¬ 
nistic  continuum  an  organization  falls  (Whittinghill,  2011).  Also,  per  the 
literature  (Damanpour,  1991;  Prakash  &  Gupta,  2008;  Robbins  &  Judge, 
2009;  Walker  2007),  these  attributes  have  a  causal  relationship  with  an 
innovative  climate.  The  literature  also  states  that  the  resource  supply 
for  innovation  attribute  has  a  causal  relationship  and  contributes  to  an 
innovative  climate  (Robbins  &  Judge,  2009;  Ruiz-Moreno  et  al.,  2008). 

Taken  together,  support  for  innovation  and  resource  supply  for  innova¬ 
tion  define  an  organization's  affinity  for  innovativeness.  The  degree  to 
which  collaboration,  workforce  autonomy,  and  managerial  trust/work¬ 
force  enthusiasm  are  present  (or  not)  determines  whether  an  organic  or 
a  mechanistic  culture  is  present,  and  subsequently  how  it  influences  an 
innovative  climate. 

Since  support  for  innovation  is  most  closely  related  to  an  innovative 
climate,  the  other  attributes  were  theorized,  supported  by  the  previ¬ 
ously  cited  research  literature,  to  influence  directly  an  organization’s 
support  for  innovation.  This  theory  was  successfully  tested  utilizing  a 
mathematical  technique  called  structural  equation  modeling 
(Whittinghill,  2011). 

Creating  an  Innovative  Organization 

Structural  equation  modeling,  as  depicted  in  Figure  1,  was  employed 
to  estimate  attribute  influence  and  theorize  attribute  relationships 
(Bollen,  1989).  It  provided  an  effective  technique  for  quantitative  analy¬ 
sis,  based  on  a  premise  that  determines  to  what  level  an  organization 
supports  innovation,  and  subsequently  an  innovative  climate.  The  prem¬ 
ise  is  influenced  by  three  primary  factors: 

1.  An  organization’s  position  on  the  organic/mechanistic 
continuum; 

2 .  An  organization’s  commitment  to  resourcing  for  innovation; 
and 
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3.  Specific  aspects  of  support  for  innovation  represented  only 
by  manifest  variables  (made  up  of  POCalCAT  Revision  A 
questions). 

Additionally,  structural  equation  modeling  provided  insight  into  the 
relationships  between  attributes  that  contribute  to  an  innovative  climate 
(i.e.,  the  independent  latent  variables).  The  attributes  modeled  were  the 
five  attributes  previously  listed.  The  manifest  variables  (i.e.,  indica¬ 
tors)  used  were  the  questions  of  the  POCalCAT  Revision  A  (which  were 
grouped  according  to  the  attributes  they  represent).  Based  on  the  causal 
relationships  found  in  the  literature  review,  a  structural  equation  model 
was  developed. 
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FIGURE  1.  STRUCTURAL  EQUATION  MODEL  FOR  POCalCAT 
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The  derived  structural  equation  model  fit  the  data  collected  by  the 
POCalCAT  Revision  A  relatively  well.  This  model  produced  an  accept¬ 
able  Root  Mean  Square  Error  of  Approximation  (RMSEA)  value  of  0.076 
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(Blunch,  2008;  Byrne,  2010),  an  acceptable  goodness  of  fit  index  of  0.797 
(Kline,  2011),  and  an  acceptable  comparative  fit  index  of  0.881  (Byrne, 
2010;  Kline,  2011),  indicating  a  relatively  good  fit. 

With  model  data  fit  established,  the  regression  weights  were  reviewed 
(Table  4).  All  modeled  relationships  (displayed  in  Figure  1)  between 
principal  components  were  statistically  significant  and  positive. 


TABLE  4.  STRUCTURAL  EQUATION  MODELING  REGRESSION 

WEIGHTS 


Latent  Variable 

Latent  Variable 

Regression 

Weight  Estimate 

Standard  Error 

Critical  Ratio 

p-value 

Support  for 

Resource  Supply 

1.87 

.553 

3.39 

<.001 

Innovation 

for  Innovation 

Support  for 

<- 

Collaboration 

.688 

.127 

5.412 

<.001 

Innovation 

Support  for 

<— 

Workforce 

.266 

.096 

2.764 

.006 

Innovation 

Autonomy 

Workforce 

<- 

Managerial  Trust/ 

.798 

.092 

8.642 

<.001 

Autonomy 

Workforce 

Enthusiasm 

For  the  latent  variables  (i.e.,  attributes)  resource  supply  for  innovation, 
collaboration,  and  workforce  autonomy,  when  the  score  of  each  on  a 
7-point  Likert  scale  went  up  by  one,  the  support  for  innovation  latent 
variable  would  go  up  1.87,  0.688,  and  0.266,  respectively.  These  regres¬ 
sion  weights  (i.e.,  regression  coefficients)  predict  the  score  of  the  support 
for  innovation  attributes  (Arbuckle,  2007;  Brewerton  &  Millward,  2006; 
Montgomery  et  al.,  2006).  If  the  managerial  trust/workforce  enthusiasm 
attribute  went  up  by  one,  then  the  workforce  autonomy  latent  variable 
would  go  up  by  0.798  (and  subsequently  support  for  innovation  would 
go  up  by  0.212).  Thus,  workforce  autonomy  has  an  indirect  effect  on  the 
support  for  innovation  attribute. 
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Conclusions 

For  this  research  study,  a  structural  equation  model  was  developed 
based  on  the  results  of  a  prior  research  literature  review  and  populated  with 
survey  data  from  the  DoD,  which  provided  the  basis  for  identifying  the  mag¬ 
nitude  of  attribute  influence  on  innovativeness.  The  analysis  of  the  model 
revealed  that  attributes  influenced  innovativeness  to  varying  degrees. 

1.  Support  for  innovation  has  the  greatest  influence  on  inno¬ 
vativeness  (per  literature  review  and  successful  structural 
equation  model  using  manifest  variables). 

2.  Resource  supply  for  innovation  is  the  next  most  influential 
attribute  (from  structural  equation  modeling). 

3.  Collaboration  is  the  third  most  influential  (from  structural 
equation  modeling). 

4.  Workforce  autonomy  is  a  distant  fourth  (from  structural  equa¬ 
tion  modeling). 

5.  Managerial  trust/workforce  enthusiasm  is  the  least  influen¬ 
tial,  but  almost  as  influential  as  workforce  autonomy  (from 
structural  equation  modeling). 

Future  efforts  to  further  develop  these  attributes  within  an  organization 
should  consider  each  attribute’s  relative  influence  on  innovativeness.  Also, 
it  should  be  understood  that  results  may  vary  for  different  organizations 
and  groups. 

Before  proceeding  further,  two  quick  notes  are  warranted: 

1.  Resource  supply  for  innovation  is  extremely  influential  accord¬ 
ing  to  the  structural  equation  model.  Since  personnel  and 
funding  allocated  for  innovative  endeavors  is  expensive,  pro¬ 
viding  time  for  such  endeavors  is  the  most  practical  resource 
to  allocate. 

2.  As  shown  previously,  collaboration,  workforce  autonomy, 
and  managerial  trust/workforce  enthusiasm  (if  present  in  an 
organization)  all  have  a  positive  influence  on  innovativeness, 
although  to  diminishing  degrees. 
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Recently,  DoD’s  senior  leaders  have  promulgated  several  public  state¬ 
ments  promoting  innovation  throughout  the  DoD  workforce.  Linear 
regression  analysis  revealed  that  the  more  organic  an  organization 
perceived  itself  to  be,  the  more  it  perceived  itself  to  be  innovative. 
This  finding  suggested  that  organizations  can  improve  innovativeness 
through  culture  manipulation.  If  the  culture  does  not  encourage  innova¬ 
tion,  the  most  effective  and  practical  actions  to  be  taken  to  change  the 
organizational  culture  and  subsequently  improve  innovativeness,  in 
priority  order,  are: 

1.  Communicate  and  demonstrate  the  importance  of  creative, 
innovative  thinking. 

2.  Give  members  time  to  think  innovatively. 

3.  Allow  and  encourage  members  to  collaborate. 

4.  Allow  members  flexibility  to  approach  problems  as  they  see 
fit,  free  from  group-think. 

5.  Assign  motivating  work  and  trust  members  to  perform 
without  being  micromanaged. 

By  implementing  these  actions,  culture  within  an  organization  can  be 
modified  to  improve  its  innovativeness,  to  advance  its  ability  to  overcome 
future  and  emerging  threats,  and  to  meet  new  and  complex  challenges. 
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Review: 

How  can  system  designers  work  together  and  coordinate  action 
across  organizational  boundaries— often  including  firms,  governments, 
and  universities— and  still  ensure  the  resulting  product  is  of  the  highest 
quality?  It's  a  question  that  has  plagued  systems  engineering  from  the 
very  beginning.  In  his  great  book  Engineering  the  F-4  Phantom  II:  Parts 
into  Systems ,  the  historian  Glenn  E.  Bugos  draws  our  attention  to  this 
issue  and  shows  how  systems  engineers  have  worked  to  resolve  it.  No 
doubt,  many  readers  of  this  journal  will  need  no  introduction  to  the  F-4 
Phantom  II,  a  fighter  jet  produced  by  McDonnell  Douglas.  It  entered  pro¬ 
duction  in  1954  and,  within  the  United  States,  was  retired  from  service 
in  1996,  ironically  the  same  year  that  Bugos's  book  was  released. 

Production  of  the  jet  was  complicated,  involving  the  military  and  sev¬ 
eral  firms,  including  McDonnell  Douglas,  General  Electric,  Raytheon, 
Westinghouse,  Collins  Radio,  and  Lear  Instrument.  The  number  of 
individuals  and  organizations  involved  made  coordination  extraordi¬ 
narily  difficult.  Moreover,  the  Phantom  II  was  re-made  several  times 
throughout  its  long  career.  As  Bugos  writes,  “The  Phantom  was  built  by 
integrating  parts  into  systems,  then  disaggregating  these  systems  into 
smaller  parts,  and  reintegrating  them  again  in  different  ways.”  This 
making,  remaking,  and  rearranging  was  true  not  just  for  the  technolo¬ 
gies,  but  also  for  the  organizations  involved,  many  of  which  went  through 
significant  transformations  during  the  technology  s  lifespan. 

Bugos  brings  the  best  aspects  of  the  field  of  science  and  technology 
studies  to  bear  on  his  subject.  While  he  spends  a  great  deal  of  time  and 
energy  spelling  out  the  formal  organizational  structures  that  were  built 
to  manage  the  Phantom  II,  he  points  out  that,  really,  the  most  important 
resource  was  trust.  This  focus  is  probably  Bugos’s  greatest  contribution 
to  the  literature  on  systems  engineering.  Interorganizational  coopera¬ 
tion  could  sometimes  break  down,  leading  to  hostility  and  competition. 
But  teams  involved  in  designing  and  managing  the  Phantom  II  created 
testing  practices,  verification  routines,  and  other  mechanical  or  quan¬ 
titative  systems  of  trust-building,  which  assured  that  everyone  was  on 
the  same  page  and  that  systems  would  operate.  In  practical  terms  then, 
Bugos  reminds  systems  engineers  that,  if  they  want  to  be  truly  success¬ 
ful,  they  must  spend  as  much  care  creating  healthy  interpersonal  and 
interorganizational  ties  as  they  do  attending  to  the  technical  dimensions 
of  their  work.  It's  a  lesson  worth  remembering. 


Defense  ARJ.  April  2015 ,  Vol  22 No.  2: 240-241 


241 


Defense  ARJ  Guidelines 

FOR  CONTRIBUTORS 

The  Defense  Acquisition  Research  Journal  (ARJ)  is  a  scholarly  peer- 
reviewed  journal  published  by  the  Defense  Acquisition  University  (DAU). 
All  submissions  receive  a  blind  review  to  ensure  impartial  evaluation. 


IN  GENERAL 

We  welcome  submissions  from  anyone  involved  in  the  defense  acqui¬ 
sition  process.  Defense  acquisition  is  defined  as  the  conceptualization, 
initiation,  design,  development,  testing,  contracting,  production,  deploy¬ 
ment,  logistics  support,  modification,  and  disposal  of  weapons  and  other 
systems,  supplies,  or  services  needed  for  a  nation’s  defense  and  security, 
or  intended  for  use  to  support  military  missions. 

Research  involves  the  creation  of  new  knowledge.  This  generally  requires 
using  material  from  primary  sources,  including  program  documents, 
policy  papers,  memoranda,  surveys,  interviews,  etc.  Articles  are  charac¬ 
terized  by  a  systematic  inquiry  into  a  subject  to  discover/revise  facts  or 
theories  with  the  possibility  of  influencing  the  development  of  acquisi¬ 
tion  policy  and/or  process. 

We  encourage  prospective  writers  to  coauthor,  adding  depth  to  manu¬ 
scripts.  It  is  recommended  that  a  mentor  be  selected  who  has  been 
previously  published  or  has  expertise  in  the  manuscript’s  subject. 
Authors  should  be  familiar  with  the  style  and  format  of  previous  Defense 
ARJs  and  adhere  to  the  use  of  endnotes  versus  footnotes  (refrain  from 
using  the  electronic  embedding  of  footnotes),  formatting  of  reference 
lists,  and  the  use  of  designated  style  guides.  It  is  also  the  responsibility 
of  the  corresponding  author  to  furnish  a  government  agency /employer 
clearance  with  each  submission. 


242 


Defense  ARJ,  April  2015,  Vol  22  No.  2:242-247 


MANUSCRIPTS 

Manuscripts  should  reflect  research  of  empirically  supported  experi¬ 
ence  in  one  or  more  of  the  areas  of  acquisition  discussed  above.  Empirical 
research  findings  are  based  on  acquired  knowledge  and  experience 
versus  results  founded  on  theory  and  belief.  Critical  characteristics  of 
empirical  research  articles: 

•  clearly  state  the  question, 

•  define  the  methodology, 

•  describe  the  research  instrument, 

•  describe  the  limitations  of  the  research, 

•  ensure  results  are  quantitative  and  qualitative, 

•  determine  if  the  study  can  be  replicated,  and 

•  discuss  suggestions  for  future  research  (if  applicable). 

Research  articles  may  be  published  either  in  print  and  online,  or  as 
a  Web-or_ly  version.  Articles  that  are  4,500  words  or  less  (excluding 
abstracts,  references,  and  endnotes)  will  be  considered  for  print  as  well 
as  Web  publication.  Articles  between  4,500  and  10,000  words  will  be 
considered  for  Web-only  publication,  with  an  abstract  (150  words  or 
less)  included  in  the  print  version  of  the  Defense  ARJ.  In  no  case  should 
article  submissions  exceed  10,000  words. 


2^3 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Book  Reviews 

Defense  ARJ  readers  are  encouraged  to  submit  reviews  of  books  they 
believe  should  be  required  reading  for  the  defense  acquisition  profes¬ 
sional.  The  reviews  should  be  400  words  or  fewer  describing  the  book 
and  its  major  ideas,  and  explaining  why  it  it  relevant  to  defense  acquisi¬ 
tion.  In  general,  book  reviews  should  reflect  specific  in-depth  knowledge 
and  understanding  that  is  uniquely  applicable  to  the  acquisition  and  life 
cycle  of  large  complex  defense  systems  and  services. 

Audience  and  Writing  Style 

The  readers  of  the  Defense  ARJ  are  primarily  practitioners  within 
the  defense  acquisition  community.  Authors  should  therefore  strive  to 
demonstrate,  clearly  and  concisely,  how  their  work  affects  this  com¬ 
munity.  At  the  same  time,  do  not  take  an  overly  scholarly  approach  in 
either  content  or  language. 

Format 

Please  submit  your  manuscript  with  references  in  APA  format 
(author-date-page  number  form  of  citation)  as  outlined  in  the  Publication 
Manual  of  the  American  Psychological  Association  (6th  Edition).  For  all 
other  style  questions,  please  refer  to  the  Chicago  Manual  of  Style  (16th 
Edition). 

Contributors  are  encouraged  to  seek  the  advice  of  a  reference  librarian 
in  completing  citation  of  government  documents  because  standard  for¬ 
mulas  of  citations  may  provide  incomplete  information  in  reference  to 
government  works.  Helpful  guidance  is  also  available  in  The  Complete 
Guide  to  Citing  Government  Documents  (Revised  Edition):  A  Manual 
for  Writers  and  Librarians  (Garner  &  Smith,  1993),  Bethesda,  MD: 
Congressional  Information  Service. 

Pages  should  be  double-spaced  and  organized  in  the  following  order: 
title  page  (titles,  12  words  or  less),  abstract  (150  words  or  less  to  conform 
with  formatting  and  layout  requirements  of  the  publication),  two- 
line  summary,  list  of  keywords  (five  words  or  less),  body  of  the  paper, 
reference  list  (only  include  works  cited  in  the  paper),  author's  note  or 
acknowledgments  (if  applicable),  and  figures  or  tables  (if  any). 

Figures  or  tables  should  not  be  inserted  or  embedded  into  the  text,  but 
segregated  (one  to  a  page)  at  the  end  of  the  text.  When  material  is  sub¬ 
mitted  electronically,  each  figure  or  table  should  be  saved  to  a  separate , 


244 


April  2015 


exportable  file  (i.e.,  a  readable  EPS  file).  For  additional  information  on 
the  preparation  of  figures  or  tables,  refer  to  the  Scientific  Illustration 
Committee,  1988,  Illustrating  Science:  Standards  for  Publication , 
Bethesda,  MD:  Council  of  Biology  Editors,  Inc.  Restructure  briefing 
charts  and  slides  to  look  similar  to  those  in  previous  issues  of  the  Defense 
ARJ. 

The  author  (or  corresponding  author  in  cases  of  multiple  authors)  should 
attach  a  signed  cover  letter  to  the  manuscript  that  provides  all  of  the 
authors'  names,  mailing  and  e-mail  addresses,  as  well  as  telephone  and 
fax  numbers.  The  letter  should  verify  that  the  submission  is  an  original 
product  of  the  author(s);  that  all  the  named  authors  materially  con¬ 
tributed  to  the  research  and  writing  of  the  paper;  that  the  submission 
has  not  been  previously  published  in  another  journal  (monographs  and 
conference  proceedings  serve  as  exceptions  to  this  policy  and  are  eligible 
for  consideration  for  publication  in  the  Defense  ARJ);  and  that  it  is  not 
under  consideration  by  another  journal  for  publication.  Details  about 
the  manuscript  should  also  be  included  in  the  cover  letter:  for  example, 
title,  word  length,  a  description  of  the  computer  application  programs, 
and  file  names  used  on  enclosed  DVD/CDs,  e-mail  attachments,  or  other 
electronic  media. 


COPYRIGHT 

The  Defense  ARJ  is  a  publication  of  the  United  States  Government 
and  as  such  is  not  copyrighted.  Because  the  Defense  ARJ  is  posted  as 
a  complete  document  on  the  DAU  homepage,  we  will  not  accept  copy¬ 
righted  manuscripts  that  require  special  posting  requirements  or 
restrictions.  If  we  do  publish  your  copyrighted  article,  we  will  print  only 
the  usual  caveats.  The  work  of  federal  employees  undertaken  as  part  of 
their  official  duties  is  not  subject  to  copyright  except  in  rare  cases. 

Web-only  publications  will  be  held  to  the  same  high  standards  and  scru¬ 
tiny  as  articles  that  appear  in  the  printed  version  of  the  journal  and  will 
be  posted  to  the  DAU  Web  site  at  www.dau.mil. 


245 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


In  citing  the  work  of  others,  please  be  precise  when  following  the  author- 
date-page  number  format.  It  is  the  contributor’s  responsibility  to  obtain 
permission  from  a  copyright  holder  if  the  proposed  use  exceeds  the 
fair  use  provisions  of  the  law  (see  U.S.  Government  Printing  Office, 
1994,  Circular  92:  Copyright  Law  of  the  United  States  of  America,  p.  15, 
Washington,  D.C.).  Contributors  will  be  required  to  submit  a  copy  of  the 
writer’s  permission  to  the  managing  editor  before  publication. 

We  reserve  the  right  to  decline  any  article  that  fails  to  meet  the 
following  copyright  requirements: 

•  The  author  cannot  obtain  permission  to  use  previously 
copyrighted  material  (e.g.,  graphs  or  illustrations)  in  the 
article. 

•  The  author  will  not  allow  DAU  to  post  the  article  in  our 
Defense  ARJ issue  on  our  Internet  homepage. 

•  The  author  requires  that  usual  copyright  notices  be  posted 
with  the  article. 

•  To  publish  the  article  requires  copyright  payment  by  the 
DAU  Press. 


SUBMISSION 

All  manuscript  submissions  should  include  the  following: 

•  Cover  letter 

•  Author  checklist 

•  Biographical  sketch  for  each  author  (70  words  or  less) 

•  Headshot  for  each  author  should  be  saved  to  a  CDR  disk  or 
e-mailed  at  300  dpi  (dots  per  inch)  or  as  a  high-print  qual¬ 
ity  JPEG  or  Tiff  file  saved  at  no  less  than  5x7  with  a  plain 
background  in  business  dress  for  men  (shirt,  tie,  and  jacket) 
and  business  appropriate  attire  for  women.  All  active  duty 
military  should  submit  headshots  in  Class  A  uniforms. 
Please  note:  images  from  Web,  Microsoft  PowerPoint,  or 
Word  will  not  be  accepted  due  to  low  image  quality. 


246 


April  2015 


•  One  copy  of  the  typed  manuscript,  including: 

0  Title  (12  words  or  less) 

0  Abstract  of  article  (150  words  or  less) 

0  Two  -line  summary 

0  Keywords  (5  words  or  less) 

°  Document  excluding  abstract  and  references  (4,500 
words  or  less  for  the  printed  edition  and  10,000 
words  or  less  for  the  online-only  content) 

These  items  should  be  sent  electronically,  as  appropriately  labeled 
files,  to  Defense  ARJ  Managing  Editor,  Norene  Fagan-Blanch  at: 
norene.fagan-blanch@dau.mil. 


247 


( 


Defense  ARJ 

PRINT  SCHEDULE 


The  Defense  ARJ  is  published  in  quarterly  theme  editions.  All  sub¬ 
missions  are  due  by  the  first  day  of  the  month.  See  print  schedule  below. 


Author  Deadline 

Issue 

July 

January 

November 

April 

January 

July 

April 

October 

In  most  cases,  the  author  will  be  notified  ~hat  the  submission  has  been 
received  within  48  hours  of  its  arrival.  Following  an  initial  review, 
submissions  will  be  referred  to  peer  reviewers  and  for  subsequent  con¬ 
sideration  by  the  Executive  Editor,  Defense  ARJ. 


Defeve ARJ,  April 2015,  Vol.  22 No.  2:248-249 


Contrib  ators  may  direct  their  questions  to  the  Managing  Editor,  Defense 
ARJ,  at  the  address  shown  below,  cr  by  calling  703-805-3801  (fax:  703- 
805-2917),  or  via  the  Internet  at  r.orene.fagan-blanch@dau.mil 


The  DAU  Homepage  can  be  accessed  at: 
h  itp://ww  w.  da  u.mil 


DEPARTMENT  OF  DEFENSE 
DEFENSE  AO  0UISITION  UNIVERSITY 
ATTN:  DAU  PRESS  (Defense  ARJ) 

9820  3ELV0IR  RD  STE  3 
FORT  BELVOIR,  VA  22060-5565 


249 


Defense  Acquisition  University 

WEB  SITE 

http:/toww.dau.mil 


Your  Online  Access  to  Acquisition  Research ,  Consulting , 
Information ,  and  Course  Offerings 


iCatalog 


Training 


•  Apply  for  Course 

•  Training  Courses 

•  Continuous  Learning  Modules 

•  Certification  Standards 

•  Targeted  Training 

•  Student  Policies  &  Mission  Assistance 

•  Student  Information  System 

•  Training  FAQs 

•  Request  a  Transcript 

Other  Products 

•  Mission  Assistance 

•  Knowledge  Sharing 

•  Research  Center 

•  DAU  Knowledge  Repository^  Acker  Archive 

•  Better  Buying  Power 

•  Life  Cycle  Chart 

•  iTunes 

•  Defense  Acquisition  Guidebook 


•  Course  Schedule 

•  Equivalency  Fulfillment 

•  Predecessors/Prerequisites 

•  Schedules 

•  Course  Material 

Publications 

•  Defense  AT&L  Magazine 

•  Defense  Acquisition  Research  Journal 

•  DAU  Brochures 

•  Archived  Catalogs 

•  Annual  Report 

•  Online  Publications 

Now  you  can  search 
the  DAU  Web  site  and 
our  online  publications! 


Defense  Acquisition  Research  Journal 

SURVEY 

Please  rate  this  publication  based  on  the  following  scores: 

5  -  Exceptional  4  -  Great  3  -  Good  2  -  Fair  1  -  Poor 

1)  How  would  you  rate  the  overall  publication? _ 

2)  How  would  you  rate  the  design  of  the  publication?  _ 

3)  Please  list  all  that  apply: _ 


True  False 

a)  This  publication  is  easy  to  read 

b)  This  publication  is  useful  to  my  career 

c)  This  publication  contributes  to  my  job  effectiveness 

d)  I  read  most  of  this  publication 

e)  I  recommend  this  publication  to  others  in  the  acquisition  field 

4)  What  themes  or  topics  would  you  like  to  see  covered  in  future  ARJsl 


5)  What  topics  would  you  like  to  see  get  less  coverage  in  future  ARJsl 


6)  How  can  we  improve  this  publication?  Provide  any  constructive  criticism  to  help  us  to 
improve  this  publication: 


7)  Please  specify  your  organization: 


NO  POSTAGE 
NECESSARY 
IF  MAILED 
IN  THE 

UNITED  STATES 


BUSINESS  REPLY  MAIL 

FIRST-CLASS  MAIL  PERMIT  NO.  12  FORT  BELVOIR, VA 
POSTAGE  WILL  BE  PAID  BY  ADDRESSEE 


DEFENSE  ACQUISITION  UNIVERSITY 

ATTN:  DAU  PRESS 
9820  BELVOIR  RD  STE  3 
FORT  BELVOIR  VA  22060-9910 


FREE 

ONLINE 

SUBSCRIPTION 


I  I  Defense  ARJ  Defense  A  T&L  O 


Thank  you  for  your  interest  in  Defense  AT&L  magazine  and  Defense  Acquisition  Research 
Journal.  To  receive  your  complimentary  online  subscription,  please  answer  all  questions 
below— incomplete  forms  cannot  be  processed. 

*When  registering,  please  do  not  include  your  rank,  grade,  service,  or  other  personal  identifiers. 


I  |  New  Online 
Subscription 


|  |  Cancellation 
I  I  Change  of  E-mail  Address 


Date 

Last  Name:  _ 

First  Name:  _ 

Day/ Work  Phone: _ 

E-mail  Address:  _ _ _ 

Signature:  (Required)  _ _ 


PLEASE  FAX  TO:  703-805-2917 

The  Privacy  Act  and  Freedom  of  Information  Act 

In  accordance  with  the  Privacy  Act  and  Freedom  of  Information  Act,  we  will  only 
contact  you  regarding  your  Defense  ARJ  and  Defense  AT&L  subscription.  If  you  provide  us 
with  your  business  e-mail  address,  you  may  become  part  of  a  mailing  list  we  are  required 
to  provide  to  other  agencies  who  request  the  lists  as  public  information.  If  you  prefer  not 
to  be  part  of  these  lists,  please  use  your  personal  e-mail  address. 


SUBSCRIPT 

1 

ver  11/28/11 


NO  POSTAGE 
NECESSARY 
IF  MAILED 
IN  THE 

UNITED  STATES 


BUSINESS  REPLY  MAIL 

FIRST-CLASS  MAIL  PERMIT  NO.  12  FORT  BELVOIR, VA 
POSTAGE  WILL  BE  PAID  BY  ADDRESSEE 


DEFENSE  ACQUISITION  UNIVERSITY 

ATTN:  DAU  PRESS 
9820  BELVOIR  RD  STE  3 
FORT  BELVOIR  VA  22060-9910 


I..I.I..I.III....II..II...I.I..I.I . INI . Ill 


Defense  Acquisition  Research  Journal 

A  Publication  of  the  Defense  Acquisition  University 


We’re  on  the  Web  at: 

http://www.dau.mil/pubscats/Pages/ARJ.aspx 


Articles  represent  the  views  of  the  authors  and  do  not  necessarily  reflect  the 
opinion  of  DAU  or  the  Department  of  Defense, 


ai**‘*» 


Defense  Acquisition  Research  Journal 

A  Publication  of  the  Deferse  Acquisition  University 


Learn.  Perform.  Succeed. 


fflHBWI 9HPI.  ___JSW 

1  a" 

